With Metasploit community CTF 2020 approaching rapidly, I decided to refresh my knowledge about website penetration testing. When dealing with Web security, I use Weber Framework, my own personal proxy, whenever possible. In this post I want to introduce the basics of this nifty tool.
Weber
Weber is a console-based web proxy heavily inspired by famous projects such as Burp Suite, WebScarab, mitmproxy and others. I chose to implement my own solution because:
- I wanted deeper knowledge about the matter,
- existing tools could not have been efficiently controlled,
- I wanted specific functionality according to my exact needs.
Despite the importance of website penetration testing, Weber is not ready for production use and is under development. However, it can already do the basic stuff and some more cool things, therefore I feel the introduction would not hurt anybody.
You shall be warned – for maximum efficiency Weber uses as short commands as possible – this idea comes from radare2 reverse engineering framework. And it is really useful approach, although the learning curve is initially quite steep. Each letter usually refers to a word; I will explain the actual meaning for every command we come across (look for text in parentheses).
Initialization
After cloning the git repository you run Weber directly as ./weber. Alternatively, you can build Weber as Docker container and run it in modern way – the command is in docker.sh file, for clarity it is:
sudo docker run --rm -it -v $PWD/files:/weber/files -p 8080:8080 --user $(id -u):$(id -g) --name weber lightfaith/weber
Weber now listens on port 8080, like other common proxies. You must now configure your browser to send requests over it. I prefer to create new profile at about:profiles (if running Firefox) and install and configure FoxyProxy plugin inside. Because of this I can pretend to be hacker in one browser window and seamlessly run Youtube in another one. The new profile can also be configured so Firefox itself will not spam the proxy with unimportant stuff. Later I might recall and sum up the way to do it.
Configure FoxyProxy as HTTP proxy to 127.0.0.1:8080. If you want to use wget instead of your browser, you can force proxy by specifying it in configuration file or as arguments, for example:
wget -e use_proxy=yes -e http_proxy=127.0.0.1:8080 -e https_proxy=127.0.0.1:8080 --no-check-certificate http://example.com
I like to store that command as alias in my ~/.bashrc:
alias wgetp="wget -e use_proxy=yes -e http_proxy=127.0.0.1:8080 -e https_proxy=127.0.0.1:8080 --no-check-certificate"
If you wonder why –no-check-certificate flag is present, it is because Weber generates fake certificates on the fly in order to grant access to encrypted flows. It is not a bug, it is a feature.
That is it, if you make a request now you should see something in Weber console:
./weber [.] Loading config file... [.] interaction.timeout_warnings = False [.] overview.short_request = True [.] Bend the knee! )> 1 http://example.com:80 GET / 200 OK 1256 B
Basic control
From this point every request you make is pushed through the Weber. Request-response pair overviews will pop-out in realtime (this feature can be disabled). First you should learn how to get help. Weber is self-documented, to see help use ? command. This will show what letters commands can start with. Write a letter and a question mark and you will see more detailed possibilities. Annoying, I know, but there is very well hidden logic. You will usually use 5-10 commands anyway.
)> ?
a[?] analysis
b[?] brute-force (alias for `pwb`)
q quit
r[?] [estu] [<rrid>[:<rrid>]] print request-response overview (alias for `ro`)
s[?] show server overview
w[?] write
)> r?
r[?] [estu] [<rrid>[:<rrid>]] print request-response overview (alias for `ro`)
rD [<rrid>[:<rrid>]] delete requests/responses
rH[?] print HTML-related info
ra[?] [<rrid>[:<rrid>]] print requests-response pairs verbose
rc[?] [<rrid>[:<rrid>]] print cookies sent in requests
rd[?] [<rrid>[:<rrid>]] print request-response data
rh[?] [<rrid>[:<rrid>]] print request-response headers
ro[?] [estu] [<rrid>[:<rrid>]] print request-response overview
rp [<rrid>[:<rrid>]] print HTTP parameters
rq[?] [<rrid>[:<rrid>]] print requests verbose
rs[?] [<rrid>[:<rrid>]] print responses verbose
rt [<rrid>[:<rrid>]] print request/response timeline
Two question marks are even better! Use ?? to get some introduction to Weber or append the question marks to a command (e.g. r??) to get help on the command directly.
)> ??
Welcome! This is Weber Framework, an open-source protocol proxy.
With Weber you can see the traffic, modify it and more!
Currently, Weber will forward anything sent to '127.0.0.1:8080
...
)> r??
Use `r` or `ro` commands to get an overview of all captured
request-response pairs.
There are optional parameters 'e', 's', 't', 'u'.
...
Options
You can tune Weber with a number of options. Use wo (weber options) command to list current settings. Edit values with wos <key> <value> (weber option set) command, e.g. wos debug.flow True. You can also set default values in weber.conf file. We will deal with relevant options when needed.
)> wo
analysis.immediate True
analysis.ignored_tests ''
brute.placeholder '###'
brute.value_separator ';'
...
Grep and less
You can filter receive output with grep-like functionality. Just specify ~ character and then your desired value. For example list available debug options with wo~debug.
)> wo~debug
debug.analysis False
debug.chunks False
debug.command False
debug.config False
debug.flow False
debug.mapping False
debug.parsing False
debug.protocol False
debug.server False
debug.socket False
debug.tampering False
Use two tilde characters to do regular expression filtering.
)> wo~~(brute|spoof)
brute.placeholder '###'
brute.value_separator ';'
brute.rps 20
brute.set_separator '\n'
spoof.arguments False
At some points you will want to read the output in more sophisticated way – why not use less command for that? Syntax is simple, just append $L to you command, e.g. wo$L.
)> wo$L
analysis.immediate True
analysis.ignored_tests ''
...
tamper.requests False
tamper.responses False
~
~
(END)
Requests and responses
Let’s move to something practial now. Mostly you will use commands related to requests and responses. Use r (requests/responses) to show the entire overview.
)> r
RRID Server Request Response Size
==== ===================== ================ ============= ======
1 http://example.com:80 GET / 200 OK 1256 B
2 http://example.com:80 GET /favicon.ico 404 Not Found 1256 B
You can see RRID, which serves as an identifier to specific request-response pairs (or RRs for short). This will be used a lot. You can also see server the request is forwarded to, request method and path and corresponding response code. Lastly you can see size of the response.
Some information is hidden, for example time of response received. That can be shown if you set overview.show_time option to True or if you specify the t flag:
)> r t
Time RRID Server Request Response Size
============ ==== ===================== ================ ============= ======
08:53:10.627 1 http://example.com:80 GET / 200 OK 1256 B
08:53:11.785 2 http://example.com:80 GET /favicon.ico 404 Not Found 1256 B
You can also show overview for specific RRs only – just specify desired RRIDs. You can use comma-separated values or intervals, e.g. 1,2,5-10. This will work for most commands.
)> r 1
RRID Server Request Response Size
==== ===================== ======= ======== ======
1 http://example.com:80 GET / 200 OK 1256 B
Too many RRs on your plate? Show just the last 10 with rol (RR overview – last) command. And you can also use the grep.
If the request column is unbearably wide, force path shortening with overview.short_request option.
RR times
Sometimes it may be useful to see more exact timing of the process. Use rt (RR timing) for exactly that:
)> rt
[!] The command has no documentation.
--- #1 ---
thread_started: 2020-01-25 08:53:09.515312
request_received: 2020-01-25 08:53:09.916196
request_forwarded: 2020-01-25 08:53:09.916692
response_received: 2020-01-25 08:53:10.627400
response_forwarded: 2020-01-25 08:53:10.629773
--- #2 ---
thread_started: 2020-01-25 08:53:10.667718
request_received: 2020-01-25 08:53:11.068784
request_forwarded: 2020-01-25 08:53:11.069128
response_received: 2020-01-25 08:53:11.785801
response_forwarded: 2020-01-25 08:53:11.787585
RR details
Now let’s see the actual content of requests and responses. There are many commands to do that, the most general is ra (RR – all) that shows everything (request, response, headers, data). You will definitely want to specify RRIDs and sometimes even the less view.
)> ra 1
--- #1 ---
GET / HTTP/1.1
Host: example.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
HTTP/1.1 200 OK
Content-Encoding: gzip
Age: 417682
Cache-Control: max-age=604800
Content-Type: text/html; charset=UTF-8
Date: Tue, 25 Jan 2020 07:53:10 GMT
Content-Length: 1256
<!doctype html>
<html>
<head>
<title>Example Domain</title>
...
</body>
</html>
For request only use rq (request) command, rs (response) for response. For headers only, use rh (RR headers) command, rd (RR data) for data. Only interested in request headers? Clearly rqh (request headers), and rsd (response data) for response data and so on. See the pattern?
)> rqh 2 --- #2 --- GET /favicon.ico HTTP/1.1 Host: example.com User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0 Accept: image/webp,*/* Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate Dnt: 1 Connection: keep-alive
Use x at the end of your previous command, e.g. rsdx 1 (response data hexadecimal):
)> rsdx 1 --- #1 --- 00000000 3c21 646f 6374 7970 6520 6874 6d6c 3e0a |<!doctype html>.| 00000010 3c68 746d 6c3e 0a3c 6865 6164 3e0a 2020 |<html>.<head>. | 00000020 2020 3c74 6974 6c65 3e45 7861 6d70 6c65 | <title>Example| 00000030 2044 6f6d 6169 6e3c 2f74 6974 6c65 3e0a | Domain</title>.| ...
How to save captured RRs into a file? Use w at the end of the command and specify path and RRIDs. You will often use rsdw /tmp/a 1 (response data write) to store received binary data for further analysis. Alternatively, use rsdwa /tmp/folder (response data write auto) to auto-save all received data.
Tampering
What a proxy without tampering feature would be useful for? Short answer: not much. Weber has capabilities of interrupting, modifying and forwarding individual requests and responses and we will look at it right now.
Enable tampering of requests with rqt (request tamper) command. This will pause forwarding of received requests so you can modify them. In overview such requests will have [T] mark near them.
You can specify number of requests to be affected: rqt 5. Short command for just one request tamper is rqt1 (request tamper 1). Finally, disable forwarding interruption with rqt- (request tamper off) command. Of course similar commands exist for responses, just start with rst (response tamper).
You can filter the overview for only tampered stuff with rot (RR overview – tampered) command.
)> rqt
[.] Requests will be TAMPERED by default.
)> r
RRID Server Request Response Size
==== ===================== ================ ============= ======
1 http://example.com:80 GET / 200 OK 1256 B
2 http://example.com:80 GET /favicon.ico 404 Not Found 1256 B
5 http://example.com:80 [T] GET / ... - B
)> rot
RRID Server Request Response Size
==== ===================== ========= ======== ====
5 http://example.com:80 [T] GET / ... - B
)> rqt-
[.] Requests are no longer tampered.
Even after stopping the tampering with rqt- or rst-, requests and responses will still hang in Weber. Already tampered requests/responses must be forwarded. Finally, forward requests with rqf (request forward) command (RRID can be also specified) and responses with rsf (response forward) command.
Tampered requests/responses can be modified with rqm <RRID> (request modify) or rsm <RRID> (response modify) command. This will open the request/response in you favourite editor, currently specified in edit.command option. See the process in asciinema at the end of this post.
Can you forward request that is not tampered? Yes, use rqr <RRID> (request replay) command to resend the request.
Save progress
Finally, you can save your entire Weber session with ww /tmp/a.web (weber write) command and restore it later with ./weber --restore /tmp/a.web. This functionality is currently tested.
Demo time
In the following asciinema features mentioned in the post are demonstrated. Namely:
- Weber help (0:09),
- options (0:28),
- grep&less (0:37),
- request/response overview (1:02),
- request/response timing (1:31),
- request and response details (1:34),
- data saving (2:21),
- request tampering, modification and forwarding (2:39),
- progress saving and restoration (3:15).
Conclusion
In this post Weber Framework proxy was introduced. Although it may seem annoying at first, you can quickly realize it offers an effective approach to help you with website security testing. Next time we will deal with some real usage. Meanwhile remember that:
- I am using RR abreviation a lot to talk about request and responses,
- Weber is (will be) self-documented – use
?to see commands and??to read more about them, - you can grep output with
~and~~modifiers, - you can read output in less with
$Lmodifier, - RR overview is evoked with
rcommand and you will use it a lot, - RR details can be retrieved with commands
ra,rq,rsand their more detailed modifications, - you can get hexdump of RRs by appending
xto previously mentioned commands and export the data by appendingw, - you can auto-save all received data with
rsdwacommand, - you can tamper, modify and forward requests with
rqt,rqm,rqfcommands, that applies to responses as well, - you can save your Weber session with
wwcommand.