Skip to content

requests.md

Latest commit

 

History

History
1739 lines (958 loc) · 28.6 KB

requests.md

File metadata and controls

1739 lines (958 loc) · 28.6 KB

Requests

Body

The body of a Request or Response.

Calling to<FORMAT> will try to convert the body to the desired format.

Constructors

Constructor

new Body(data: string | number[] | Uint8Array): Body

Parameters
Parameter Type
data string | number[] | Uint8Array
Returns

Body

Properties

length

readonly length: number

The length of the body in bytes.

Methods

toJson()

toJson(): unknown

Try to parse the body as JSON.

Returns

unknown

Throws

If the body is not valid JSON.

toRaw()

toRaw(): Uint8Array

Get the raw body as an array of bytes.

Returns

Uint8Array

toText()

toText(): string

Parse the body as a string.

Unprintable characters will be replaced with �.

Returns

string

RequestSpec

A mutable Request that has not yet been sent.

Constructors

Constructor

new RequestSpec(url: string): RequestSpec

Build a new RequestSpec from a URL string. We try to infer as much information as possible from the URL, including the scheme, host, path and query.

You can convert a saved immutable Request object into a RequestSpec object by using the toSpec() method.

By default:

  • Method is GET.
  • Path is /.
Parameters
Parameter Type
url string
Returns

RequestSpec

Throws

If the URL is invalid.

Example
const spec = new RequestSpec("https://example.com");

Methods

getBody()

getBody(): Body | undefined

The body of the request.

Returns

Body | undefined

getHeader()

getHeader(name: string | GetHeaderOptions): string[] | undefined

Get a header value.

Header name is case-insensitive. The header might have multiple values.

Parameters
Parameter Type
name string | GetHeaderOptions
Returns

string[] | undefined

getHeaders()

getHeaders(): Record<string, string[]>

The headers of the request.

Header names are case-insensitive. Each header might have multiple values.

Returns

Record<string, string[]>

Example
{
  "Host": ["caido.io"],
  "Connection": ["keep-alive"],
  "Content-Length": ["95"]
}
getHost()

getHost(): string

Get the host of the request.

Returns

string

getInfo()

getInfo(): ConnectionInfo

Get the connection information of the request.

Returns

ConnectionInfo

The connection information.

getMethod()
Call Signature

getMethod(): string

Get the HTTP method of the request.

Get the raw version by passing { raw: true } in the options.

Returns

string

Call Signature

getMethod(options: RawOption): Uint8Array

Get the HTTP method of the request.

Get the raw version by passing { raw: true } in the options.

Parameters
Parameter Type
options RawOption
Returns

Uint8Array

getPath()
Call Signature

getPath(): string

Get the path of the request.

Get the raw version by passing { raw: true } in the options.

Returns

string

Call Signature

getPath(options: RawOption): Uint8Array

Get the path of the request.

Get the raw version by passing { raw: true } in the options.

Parameters
Parameter Type
options RawOption
Returns

Uint8Array

getPort()

getPort(): number

Get the port of the request.

Returns

number

getQuery()
Call Signature

getQuery(): string

Get the unparsed query of the request.

Get the raw version by passing { raw: true } in the options.

Excludes the leading ?.

Returns

string

Call Signature

getQuery(options: RawOption): Uint8Array

Get the unparsed query of the request.

Get the raw version by passing { raw: true } in the options.

Excludes the leading ?.

Parameters
Parameter Type
options RawOption
Returns

Uint8Array

getRaw()

getRaw(): RequestSpecRaw

This methods converts the RequestSpec to a RequestSpecRaw.

This is useful to retrieve the raw bytes of the request.

Returns

RequestSpecRaw

Example
const spec = new RequestSpec("https://example.com");
const specRaw = spec.getRaw();
const bytes = specRaw.getRaw(); // GET / HTTP/1.1\r\nHost: example.com\r\n\r\n
getTls()

getTls(): boolean

Get if the request uses TLS (HTTPS).

Returns

boolean

getUrl()

getUrl(): string

The full URL of the request.

Returns

string

removeHeader()

removeHeader(name: string): void

Removes a header.

Parameters
Parameter Type
name string
Returns

void

setBody()

setBody(body: Bytes | Body, options?: SetBodyOptions): void

Set the body of the request.

The body can either be a Body or any type that can be converted to Bytes.

Parameters
Parameter Type
body Bytes | Body
options? SetBodyOptions
Returns

void

Example
const body = new Body("Hello world.");
const options = { updateContentLength: true };
request.setBody(body, options);
setHeader()

setHeader(name: string, value: string): void

Set a header value.

This will overwrite any existing values.

Parameters
Parameter Type
name string
value string
Returns

void

setHost()

setHost(host: string): void

Set the host of the request.

It will also update the Host header.

Parameters
Parameter Type
host string
Returns

void

setMethod()

setMethod(method: Bytes): void

Set the HTTP method of the request.

All strings are accepted.

Parameters
Parameter Type
method Bytes
Returns

void

setPath()

setPath(path: Bytes): void

Set the path of the request.

Parameters
Parameter Type
path Bytes
Returns

void

setPort()

setPort(port: number): void

Set the port of the request.

The port number must be between 1 and 65535.

Parameters
Parameter Type
port number
Returns

void

setQuery()

setQuery(query: Bytes): void

Set the unparsed query of the request.

The query string should not include the leading ?.

Parameters
Parameter Type
query Bytes
Returns

void

Example
spec.setQuery("q=hello");
setRaw()

setRaw(raw: Bytes): RequestSpecRaw

This method sets the raw Bytes of the request and converts it to a RequestSpecRaw.

This is useful when you have a prepared RequestSpec and you just want to modify the raw data.

Parameters
Parameter Type
raw Bytes
Returns

RequestSpecRaw

Example
const rawBytes = []; // RAW BYTES HERE
const request = new RequestSpec("https://example.com");
const rawRequest = request.setRaw(rawBytes);
setTls()

setTls(tls: boolean): void

Set if the request uses TLS (HTTPS).

Parameters
Parameter Type
tls boolean
Returns

void

parse()
Call Signature

static parse(bytes: Bytes): RequestSpec

Parses raw bytes into a RequestSpec.

Parameters
Parameter Type
bytes Bytes
Returns

RequestSpec

Throws

If the bytes are not a valid HTTP request.

Example
const rawInput = 'GET / HTTP/1.1\r\nHost: example.com\r\n\r\n';
const spec = RequestSpec.parse(rawInput);
spec.setHeader('x-caido', 'test');
const specRaw = spec.getRaw();
const rawOutput = specRaw.getRaw(); // Will contain the new header
Call Signature

static parse(raw: RequestSpecRaw): RequestSpec

Parses the raw bytes of a RequestSpecRaw into a RequestSpec.

Parameters
Parameter Type
raw RequestSpecRaw
Returns

RequestSpec

Throws

If the bytes are not a valid HTTP request.

RequestSpecRaw

A mutable raw Request that has not yet been sent.

Constructors

Constructor

new RequestSpecRaw(url: string): RequestSpecRaw

Build a new RequestSpecRaw from a URL string. Only the host, port and scheme will be parsed.

You can convert a saved immutable Request object into a RequestSpecRaw object by using the toSpecRaw() method.

You MUST use setRaw to set the raw bytes of the request.

Parameters
Parameter Type
url string
Returns

RequestSpecRaw

Example
const spec = new RequestSpecRaw("https://example.com");

Methods

getHost()

getHost(): string

Get the host of the request.

Returns

string

getInfo()

getInfo(): ConnectionInfo

Get the connection information of the request.

Returns

ConnectionInfo

The connection information.

getPort()

getPort(): number

Get the port of the request.

Returns

number

getRaw()

getRaw(): Uint8Array

Get the raw bytes of the request.

Returns

Uint8Array

getSpec()

getSpec(): RequestSpec

This methods converts the RequestSpecRaw to a RequestSpec.

Returns

RequestSpec

Deprecated

Use toSpec instead.

Throws

If the bytes are not a valid HTTP request.

See

RequestSpec.parse

getTls()

getTls(): boolean

Get if the request uses TLS (HTTPS).

Returns

boolean

setHost()

setHost(host: string): void

Set the host of the request.

It will NOT update the Host header.

Parameters
Parameter Type
host string
Returns

void

setPort()

setPort(port: number): void

Set the port of the request.

The port number must be between 1 and 65535.

Parameters
Parameter Type
port number
Returns

void

setRaw()

setRaw(raw: Bytes): void

Set the raw Bytes of the request.

Parameters
Parameter Type
raw Bytes
Returns

void

setTls()

setTls(tls: boolean): void

Set if the request uses TLS (HTTPS).

Parameters
Parameter Type
tls boolean
Returns

void

toSpec()

toSpec(): RequestSpec

This methods converts the RequestSpecRaw to a RequestSpec.

Returns

RequestSpec

Throws

If the bytes are not a valid HTTP request.

See

RequestSpec.parse

GetHeaderOptions

GetHeaderOptions = object

Options for getting a header value.

Properties

name

name: string

The name of the header to get. Header name is case-insensitive.

split?

optional split: boolean

Whether to split the header value on commas.

Default
false

Request

Request = object

An immutable saved Request.

To modify, use toSpec to get a RequestSpec object.

Methods

getBody()

getBody(): Body | undefined

The body of the request.

Returns

Body | undefined

getCreatedAt()

getCreatedAt(): Date

The datetime the request was recorded by the proxy.

Returns

Date

getHeader()

getHeader(name: string | GetHeaderOptions): string[] | undefined

Get a header value.

Header name is case-insensitive. The header might have multiple values.

Parameters
Parameter Type
name string | GetHeaderOptions
Returns

string[] | undefined

getHeaders()

getHeaders(): Record<string, string[]>

The headers of the request.

Header names are case-insensitive. Each header might have multiple values.

Returns

Record<string, string[]>

Example
{
  "Host": ["caido.io"],
  "Connection": ["keep-alive"],
  "Content-Length": ["95"]
}
getHost()

getHost(): string

The target host of the request.

Returns

string

getId()

getId(): ID

The unique Caido ID of the request.

Returns

ID

getMethod()

getMethod(): string

The HTTP method of the request.

Returns

string

getPath()

getPath(): string

The path of the request.

Returns

string

getPort()

getPort(): number

The target port of the request.

Returns

number

getQuery()

getQuery(): string

The unparsed query of the request.

Excludes the leading ?.

Returns

string

getRaw()

getRaw(): RequestRaw

The raw version of the request.

Used to access the bytes directly.

Returns

RequestRaw

getTls()

getTls(): boolean

If the request uses TLS (HTTPS).

Returns

boolean

getUrl()

getUrl(): string

The full URL of the request.

Returns

string

toSpec()

toSpec(): RequestSpec

Copied the request to a mutable un-saved RequestSpec. This enables you to make modify a request before re-sending it.

Returns

RequestSpec

toSpecRaw()

toSpecRaw(): RequestSpecRaw

Copied the request to a mutable un-saved RequestSpecRaw. The raw requests are not parsed and can be used to send invalid HTTP Requests.

Returns

RequestSpecRaw

RequestOrderField

RequestOrderField = "ext" | "host" | "id" | "method" | "path" | "query" | "created_at" | "source"

Field to order requests by.

RequestRaw

RequestRaw = object

An immutable saved raw Request.

Methods

toBytes()

toBytes(): Uint8Array

Get the raw request as an array of bytes.

Returns

Uint8Array

toText()

toText(): string

Parse the raw request as a string.

Unprintable characters will be replaced with �.

Returns

string

RequestResponse

RequestResponse = object

An immutable saved Request and Response pair.

Properties

request

request: Request

response

response: Response

RequestResponseOpt

RequestResponseOpt = object

An immutable saved Request and optional Response pair.

Properties

request

request: Request

response?

optional response: Response

RequestsConnection

RequestsConnection = object

A connection of requests.

Properties

items

items: RequestsConnectionItem[]

pageInfo

pageInfo: PageInfo

RequestsConnectionItem

RequestsConnectionItem = object

An item in a connection of requests.

Properties

cursor

cursor: Cursor

request

request: Request

response?

optional response: Response

RequestSendOptions

RequestSendOptions = object

Options for sending a request.

Properties

connection?

optional connection: Connection

The Connection to use for the request.

If provided, the request will be sent through the connection.

If not provided, the engine will open a new connection to the target.

Default
undefined
plugins?

optional plugins: boolean

If true, the request will be sent through the upstream plugins.

It defaults to to true most of the time except when called from a onUpstream callback.

Default
true
save?

optional save: boolean

If true, the request and response will be saved to the database and the user will see them in the Search tab.

If you do not save, the request and response IDs will be set to 0.

Default
true
timeouts?

optional timeouts: RequestSendTimeouts | number

The timeouts to use for sending a request and receiving a response.

If a number is provided, it will be used as the global timeout and the other timeouts will be set to infinity.

See the RequestSendTimeouts for the default values.

RequestSendPayload

RequestSendPayload = RequestResponse & object

A saved Request and Response pair with the connection used to send the request.

Type Declaration

connection

connection: Connection

RequestSendTimeouts

RequestSendTimeouts = object

Timeouts for sending a request and receiving a response.

Properties

connect?

optional connect: number

The timeout to open the TCP connection to the target host and perform the TLS handshake.

Defaults to 30s.

extra?

optional extra: number

The timeout to read data after we have a read the full response.

This is useful if you believe the server will send more data than implied by the Content-Length header.

Defaults to 0s (no timeout).

global?

optional global: number

The global timeout for sending a request and receiving a response.

No default value.

partial?

optional partial: number

The timeout between each read attempt for the response. On a slow connection, this is important to increase.

Defaults to 5s.

response?

optional response: number

The timeout to receive the first byte of the response.

After the first byte is received, the partial timeout will be used.

Defaults to 30s.

RequestsQuery

RequestsQuery = object

Query builder to fetch requests.

Methods

after()

after(cursor: Cursor): RequestsQuery

Requests after a given cursor.

Parameters
Parameter Type Description
cursor Cursor Cursor of the request
Returns

RequestsQuery

ascending()
Call Signature

ascending(target: "req", field: RequestOrderField): RequestsQuery

Ascending ordering.

Parameters
Parameter Type Description
target "req" Target of the ordering: req or resp.
field RequestOrderField Field to order by.
Returns

RequestsQuery

Call Signature

ascending(target: "resp", field: ResponseOrderField): RequestsQuery

Parameters
Parameter Type
target "resp"
field ResponseOrderField
Returns

RequestsQuery

before()

before(cursor: Cursor): RequestsQuery

Requests before a given cursor.

Parameters
Parameter Type Description
cursor Cursor Cursor of the request
Returns

RequestsQuery

descending()
Call Signature

descending(target: "req", field: RequestOrderField): RequestsQuery

Descending ordering.

Parameters
Parameter Type Description
target "req" Target of the ordering: req or resp.
field RequestOrderField Field to order by.
Returns

RequestsQuery

Call Signature

descending(target: "resp", field: ResponseOrderField): RequestsQuery

Parameters
Parameter Type
target "resp"
field ResponseOrderField
Returns

RequestsQuery

execute()

execute(): Promise<RequestsConnection>

Execute the query.

Returns

Promise<RequestsConnection>

Throws

If a query parameter is invalid or the query cannot be executed.

filter()

filter(filter: string): RequestsQuery

Filter requests.

Parameters
Parameter Type Description
filter string HTTPQL filter
Returns

RequestsQuery

first()

first(n: number): RequestsQuery

First n requests.

Parameters
Parameter Type Description
n number Number of requests to return
Returns

RequestsQuery

last()

last(n: number): RequestsQuery

Last n requests.

Parameters
Parameter Type Description
n number Number of requests to return
Returns

RequestsQuery

RequestsSDK

RequestsSDK = object

The SDK for the Requests service.

Methods

get()

get(id: ID): Promise<RequestResponseOpt | undefined>

Get a request by its unique ID.

Parameters
Parameter Type
id ID
Returns

Promise<RequestResponseOpt | undefined>

Example
await sdk.requests.get("1");
inScope()

inScope(request: RequestSpec | Request, scopes?: Scope[] | ID[]): boolean

Checks if a request is in scope.

Parameters
Parameter Type Description
request RequestSpec | Request The request to check
scopes? Scope[] | ID[] Optional scopes or scope IDs to check against. If not provided, checks against the default scope.
Returns

boolean

True if the request is in scope

Example
// Check against default scope
if (sdk.requests.inScope(request)) {
 sdk.console.log("In scope");
}

// Check against specific scopes
const isInScope = sdk.requests.inScope(request, [scope1, scope2]);
sdk.console.log(isInScope); // true or false
matches()

matches(filter: string, request: Request, response?: Response): boolean

Checks if a request/response matches an HTTPQL filter.

Parameters
Parameter Type Description
filter string HTTPQL filter
request Request The Request to match against
response? Response The Response to match against
Returns

boolean

query()

query(): RequestsQuery

Query requests of the current project.

Returns

RequestsQuery

Example
const page = await sqk.requests.query().first(2).execute();
sdk.console.log(`ID: ${page.items[1].request.getId()}`);
send()

send(request: RequestSpec | RequestSpecRaw, options?: RequestSendOptions): Promise<RequestSendPayload>

Sends an HTTP request, either a RequestSpec or RequestSpecRaw.

This respects the upstream proxy settings.

Parameters
Parameter Type
request RequestSpec | RequestSpecRaw
options? RequestSendOptions
Returns

Promise<RequestSendPayload>

Throws

If the request cannot be sent. If the request times out, the error message will contain the word "Timeout".

Example
const spec = new RequestSpec("https://example.com");
try {
  const res = await sdk.requests.send(request)
  sdk.console.log(res.request.getId());
  sdk.console.log(res.response.getCode());
} catch (err) {
  sdk.console.error(err);
}

Response

Response = object

An immutable saved Response.

Methods

getBody()

getBody(): Body | undefined

The body of the response

Returns

Body | undefined

getCode()

getCode(): number

The status code of the response.

Returns

number

getCreatedAt()

getCreatedAt(): Date

The datetime the response was recorded by the proxy.

Returns

Date

getHeader()

getHeader(name: string | GetHeaderOptions): string[] | undefined

Get a header value.

Header name is case-insensitive. The header might have multiple values.

Parameters
Parameter Type
name string | GetHeaderOptions
Returns

string[] | undefined

getHeaders()

getHeaders(): Record<string, string[]>

The headers of the response.

Header names are case-insensitive. Each header might have multiple values.

Returns

Record<string, string[]>

Example
{
  "Date": ["Sun, 26 May 2024 10:59:21 GMT"],
  "Content-Type": ["text/html"]
}
getId()

getId(): ID

The unique Caido ID of the response.

Returns

ID

getRaw()

getRaw(): ResponseRaw

The raw version of the response.

Used to access the bytes directly.

Returns

ResponseRaw

getRoundtripTime()

getRoundtripTime(): number

The time it took to send the request and receive the response in milliseconds.

Returns

number

ResponseOrderField

ResponseOrderField = "length" | "roundtrip" | "code"

Field to order responses by.

ResponseRaw

ResponseRaw = object

An immutable saved raw Response.

Methods

toBytes()

toBytes(): Uint8Array

Get the raw response as an array of bytes.

Returns

Uint8Array

toText()

toText(): string

Parse the raw response as a string.

Unprintable characters will be replaced with �.

Returns

string

SetBodyOptions

SetBodyOptions = object

Options when setting the body of a Request.

Properties

updateContentLength

updateContentLength: boolean

Should update the Content-export type header.

Default
true