-
-
[原创]Libc-Database 本地搭建实战
-
2020-11-7 20:42
-
0x01 前言
此处，我们以Ubuntu 16.04虚拟机为例讲述搭建方法，为防止环境冲突，请选择使用一台干净的虚拟机，若使用Ubuntu Server 16.04，可能会出现系统安装失败情况，若出现，请参考底部的参考链接。
另外，此教程中涉及到了部分文件的修改，我也对原项目进行了部分本地化，因此也可以直接获取我fork修改的项目进行搭建，项目地址：https://github.com/hebtuerror404/libc-database。欢迎Star~
本文中最后生成的配置结束的PVM虚拟机已上传至百度云盘，需要的自取即可，链接: https://pan.baidu.com/s/1mfG_ES4GzBrLyoGRerZDoA 密码: 9vpp
0x02 安装环境所需的依赖（需要 ，需要较长⏱️）
1. 更新系统内的所有程序包（需要 ，需要较长⏱️）
使用命令sudo apt-get update && sudo apt-get upgrade -y
⚠️：可以使用以下命令换源以加速下载
1 2 3 4 5 | sudo cp /etc/apt/sources.list /etc/apt/sources.list.baksudo vi /etc/apt/sources.list# vi 内运行以下两条命令:% s/us.archive.ubuntu.com/mirrors.aliyun.com/g:wq |
2. 通过apt安装必要软件包（需要 ）
使用命令
1 2 3 4 5 6 7 | sudo apt-get install git vim wget curl python python3 python-pip python3-pip binutils file wget rpm2cpio cpio zstd nginx libffi6 libffi-dev software-properties-common -ysudo add-apt-repository ppa:deadsnakes/ppasudo apt-get updatesudo apt-get install python3.6sudo curl https://bootstrap.pypa.io/get-pip.py -o /get-pip.pysudo python3.6 /get-pip.pysudo ln -s /usr/local/bin/pip /usr/bin/pip |
3. 通过pip安装必要软件包（需要 ）
使用命令
1 | sudo pip install elasticsearch |
4. 安装 docker 以及 docker-compose（需要 ）
此处推荐使用以下命令直接获取docker
1 | curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun |
docker成功安装后可以去阿里云官网获取镜像加速器，然后按照官网所述直接做如下配置即可。
1 2 3 4 5 6 7 8 | sudo mkdir -p /etc/dockersudo tee /etc/docker/daemon.json <<-'EOF'{ "registry-mirrors": ["https://<镜像加速器ID>.mirror.aliyuncs.com"]}EOFsudo systemctl daemon-reloadsudo systemctl restart docker |
然后运行sudo pip install docker-compose -i https://mirrors.aliyun.com/pypi/simple/安装docker-compose
⚠️：此处安装完成后，可能产生以下错误
此时可以使用which docker-compose来确定核心文件位置，然后建立符号链接即可。
5. 安装nodejs以及npm（需要 ）
首先运行以下命令获取nodejs
1 2 | curl -sL https://deb.nodesource.com/setup_14.x | sudo -E bash -sudo apt-get install -y nodejs |
然后安装npm
1 2 | sudo npm install -g nsudo n latest |
0x03 获取 libc-database 项目并处理
1. 获取项目。（需要 ）
使用命令git clone https://github.com/niklasb/libc-database.git获取此项目。
2. 修改项目内的Node.js源码。
修改libc-database/searchengine/frontend/src/App.js
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | import React, { useState, useEffect, useCallback, useRef } from 'react';import './App.css';import 'fontsource-roboto';import Button from '@material-ui/core/Button';import Grid from '@material-ui/core/Grid';import TextField from '@material-ui/core/TextField';import Link from '@material-ui/core/Link';import CircularProgress from '@material-ui/core/CircularProgress';import Table from '@material-ui/core/Table';import TableBody from '@material-ui/core/TableBody';import TableCell from '@material-ui/core/TableCell';import TableRow from '@material-ui/core/TableRow';import { makeStyles } from '@material-ui/core/styles';- const API_BASE = 'https://libc.rip/api';+ const API_BASE = 'http://127.0.0.1/api';const api = async (path, data) => { let resp = await fetch(`${API_BASE}${path}`, { method: 'POST', mode: 'cors', cache: 'no-cache', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(data), }); return await resp.json();};const useStyles = makeStyles((theme) => ({ root: { '& .MuiTextField-root': { margin: theme.spacing(1), }, '& .MuiButton-root': { margin: theme.spacing(1), }, '& .remove': { marginTop: '1.2rem', height: '2rem', }, '& .findbutton': { marginTop: '1.2rem', }, }, table: { marginTop: '1rem', marginBottom: '1rem', }}));function SearchRow({ onChange = () => {}, onRemove = () => {} }) { const [symbol, setSymbol] = useState(""); const [address, setAddress] = useState(""); const [addressValid, setAddressValid] = useState(true); const onSymbolChange = useCallback((evt) => { setSymbol(evt.target.value); }, []); const onAddressChange = useCallback((evt) => { setAddress(evt.target.value); }, []); useEffect(() => { const valid = !!address.match(/^(0x)?[0-9a-fA-F]*$/); setAddressValid(valid); onChange({valid, symbol, address}); }, [address, symbol, onChange]); return ( <div> <TextField label="Symbol name" value={symbol} onChange={onSymbolChange} /> <TextField label="Address" error={!addressValid} value={address} onChange={onAddressChange} /> <Button className="remove" variant="contained" color="secondary" onClick={onRemove}> Remove </Button> </div> );}function SearchForm({ onSearch = () => {} }) { const classes = useStyles(); const [nextId, setNextId] = useState(0); const [rows, setRows] = useState([]); const [states, setStates] = useState({}); const onRemoveRef = useRef(); const onChangeRef = useRef(); const makeRow = (id) => { return ( <SearchRow key={id} onRemove={() => onRemoveRef.current(id)} onChange={(obj) => onChangeRef.current(id, obj)} />); }; const isEmpty = useCallback((i) => { let state = states[rows[i].key]; return !state || (!state.symbol && !state.address); }, [rows, states]); // Add new empty rows automatically useEffect(() => { let need = true; for (let i = 0; i < rows.length; ++i) { if (isEmpty(i)) { need = false; break; } } if (need) { setRows(rows => rows.concat([makeRow('' + nextId)])); setNextId(id => id + 1); } }, [rows, states, nextId, isEmpty]); // Remove superfluous rows at the end useEffect(() => { let i = rows.length - 1; while (i >= 1 && isEmpty(i) && isEmpty(i-1)) { --i; } if (i < rows.length - 1) { setRows(rows => rows.slice(0, i+1)); } }, [rows, states, nextId, isEmpty]); const onRemove = useCallback((id) => { for (let i = 0; i < rows.length; ++i) { if (rows[i].key === id) { setRows(rows.slice(0, i).concat(rows.slice(i+1))); return; } } }, [rows]); const onChange = useCallback((id, obj) => { setStates({...states, [id]: obj}); }, [states]); onChangeRef.current = onChange; onRemoveRef.current = onRemove; const onSubmit = useCallback(() => { let symbols = {}; for (let row of rows) { let state = states[row.key]; if (state && state.valid && state.address && state.symbol) { symbols[state.symbol] = state.address; } } onSearch({"symbols": symbols}); }, [rows, states, onSearch]); const isValid = useCallback(() => { let cnt = 0; for (let row of rows) { let state = states[row.key]; if (!state) continue; if (!state.valid) return false; if (state.address && state.symbol) cnt++; } return cnt > 0; }, [rows, states]); return ( <form className={classes.root}> {rows} <div> <Button disabled={!isValid()} variant="contained" className="findbutton" color="primary" onClick={onSubmit}> Find </Button> </div> </form> );}function Result({ id, buildid, md5, symbols, download_url }) { const classes = useStyles(); const [open, setOpen] = useState(false); const onToggle = useCallback((evt) => { evt.preventDefault(); setOpen(!open); }, [open]); let symbolRows = Object.entries(symbols).map(([k, v]) => ( <TableRow key={k}> <TableCell><code>{k}</code></TableCell> <TableCell><code>{v}</code></TableCell> </TableRow> )); return ( <div> <Link href='#' onClick={onToggle}>{id}</Link> {open && ( <Table size="small" className={classes.table}> <TableBody> <TableRow> <TableCell>Download</TableCell> <TableCell> <Link href={download_url} download>Click to download</Link> </TableCell> </TableRow> <TableRow> <TableCell>BuildID</TableCell> <TableCell>{buildid}</TableCell> </TableRow> <TableRow> <TableCell>MD5</TableCell> <TableCell>{md5}</TableCell> </TableRow> {symbolRows} </TableBody> </Table> )} </div> );}function App() { const [loading, setLoading] = useState(false); const [results, setResults] = useState(null); const onSearch = (data) => { setLoading(true); (async () => { try { setResults(await api('/find', data)); } finally { setLoading(false); } })(); }; return ( <div className="App"> <p>Powered by the <Link href="https://github.com/niklasb/libc-database/tree/master/searchengine">libc-database search API</Link></p> <Grid container spacing={2}> <Grid item sm={12} md={6}> <h3>Search</h3> <SearchForm onSearch={onSearch} /> </Grid> <Grid item sm={12} md={6}> <h3>Results</h3> {loading && <CircularProgress />} {results !== null && results.map(x => <Result {...x} />)} </Grid> </Grid> </div> );}export default App; |
3. 生成此项目。（需要 ）
接下来进入项目文件夹并生成Web端主项目
1 2 3 | cd libc-database/searchengine/frontend/sudo npm installsudo npm run build |
⚠️：若npm安装过慢，可以尝试使用cnpm，官网对于cnpm的描述如下：
- 这是一个完整
npmjs.org镜像，你可以用此代替官方版本(只读)，同步频率目前为 10分钟 一次以保证尽量与官方服务同步。 - 从 registry.npm.taobao.org 安装所有模块. 当安装的时候发现安装的模块还没有同步过来, 淘宝 NPM 会自动在后台进行同步, 并且会让你从官方 NPM registry.npmjs.org 进行安装. 下次你再安装这个模块的时候, 就会直接从 淘宝 NPM 安装了.
要使用cnpm则运行以下命令：
1 2 3 4 | npm install -g cnpm --registry=https://registry.npm.taobao.orgcd libc-database/searchengine/frontend/sudo cnpm installcnpm run build |
0x04 本地获取所有libc（需要 ，需要极长⏱️）
⚠️：若可以在虚拟机内部开启VPN以加速访问境外网站，请在此步开启。
使用以下命令获取/更新所有libc文件到本地，此步骤依据网络连接状况需要耗时30 min ~ 2 h，请耐心等待
1 2 3 4 | cd libc-database/./get ubuntu debian centos kali parrotsec./get arch./get rpm |
./get arch命令可能极其缓慢，属于正常情况，请耐心等待~
./get rpm命令可能会因cpio错误而退出，属于正常情况，直接无视即可~
0x05 本地启动查询服务
1. 修改源文件
首先需要修改一系列文件，使之满足启动条件
修改libc-database/searchengine/nginx.conf
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | user www-data;worker_processes auto;pid /run/nginx.pid;include /etc/nginx/modules-enabled/*.conf;events { worker_connections 768;}http { sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream;- ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE- ssl_prefer_server_ciphers on; access_log /var/log/nginx/access.log; error_log /var/log/nginx/error.log; gzip on; server {- listen 443;+ listen 80; server_name libc.rocks;- ssl on;- ssl_certificate /etc/letsencrypt/live/libc.rip/fullchain.pem;- ssl_certificate_key /etc/letsencrypt/live/libc.rip/privkey.pem; location /api/ { proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection keep-alive; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } location /download/ {- alias /home/niklas/libc-database/db/;+ alias /home/error404/libc-database/db/; } location / {- alias /home/niklas/libc-database/searchengine/frontend/build/;+ alias /home/error404/libc-database/searchengine/frontend/build/; } }} |
修改libc-database/searchengine/config.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 | import osES_INDEX_NAME = 'libcsearch'# ES_HOST = 'localhost'ES_HOST = 'es01'# DB_DIR = os.path.dirname(os.path.abspath(__file__)) + '/../db'DB_DIR = '/db'DEFAULT_SYMBOLS = [ '__libc_start_main_ret', 'system', 'dup2', 'str_bin_sh', 'read', 'write', 'puts', 'printf',]- DOWNLOAD_URL = 'https://libc.rip/download/{}.so'+ DOWNLOAD_URL = 'https://127.0.0.1/download/{}.so' |
修改libc-database/searchengine/api.py
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | openapi: 3.0.3info: version: 1.0.0 title: libc-database search engine description: ''paths: /find: post: operationId: app.find tags: - libcsearch description: |- Look up libc by various attributes requestBody: content: application/json: schema: $ref: '#/components/schemas/Libc' required: true responses: '200': content: application/json: schema: type: array items: $ref: '#/components/schemas/Libc' description: '' /libc/{id}: post: operationId: app.dump tags: - libcsearch parameters: - in: path name: id schema: type: string required: true description: |- Dump libc symbols requestBody: content: application/json: schema: $ref: '#/components/schemas/DumpRequest' required: true responses: '200': content: application/json: schema: $ref: '#/components/schemas/Libc' description: ''components: schemas: Libc: type: object properties: md5: type: string sha1: type: string sha256: type: string buildid: type: string id: type: string symbols: type: object additionalProperties: type: string pattern: '^(0x)?[a-fA-F0-9]+$' download_url: type: string format: url readOnly: true DumpRequest: type: object properties: symbols: type: array items: type: stringservers:- - url: https://libc.rip/api+ - url: http://127.0.0.1/api |
2. 启动elasticsearch查询服务以及查询后端uwsgi服务(守护服务)
使用命令sudo docker-compose up -d
启动后可以使用docker ps检查启动情况
3. 建立elasticsearch查询服务索引
在/libc-database/searchengine下，执行python3.6 -m index ../db
建立完成后，使用curl -XGET 127.0.0.1:9200/_cat/indices/来测试索引建立情况
3. 启动本地Nginx服务。
最后，在/libc-database/searchengine下使用/usr/sbin/nginx -c nginx.conf启动本地Ngnix服务
4. 测试服务状态。
使用命令curl -X POST -H 'Content-Type: application/json' --data '{"symbols": {"strncpy": "db0", "strcat": "0x000000000d800"}}' http://127.0.0.1/api/find进行测试。
5.Enjoy It
断开虚拟机网络，以验证本地状态下的服务可用性
0x06 查询服务的更新（需要 ）
执行下列命令即可
1 2 3 4 5 | cd libc-database/./get ubuntu debian centos kali parrotsec./get arch./get rpmpython3.6 -m index ../db |
0x07 参考链接
【原】光盘安装ubuntu出现BusyBox不能继续安装的终极解决方法
[CTF入门培训]顶尖高校博士及硕士团队亲授《30小时教你玩转CTF》，视频+靶场+题目！助力进入CTF世界
