Skip to content
godblessforhimselfPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

IPv6地址空间探测工具:认为熵最低的位包含更密集的活跃地址,进而缩小探测范围,增大命中率。
Song, Guanglei, et al. "Towards the Construction of Global IPv6 Hitlist and Efficient Probing of IPv6 Address Space." 2020 IEEE/ACM 28th International Symposium on Quality of Service (IWQoS). IEEE, 2020.

sudo bash all.sh /home/tony/datas/2019-05-01/Test.txt /home/tony/datas/2019-05-01/test 2402:f000:4:38:808::382
sudo bash all.sh /home/tony/datas/2019-05-01/2019-05-01-input.txt /home/tony/datas/2019-05-01/Main 2402:f000:4:38:808::382 4
sudo python patternMining.py --experience=False
sudo bash all.sh /home/tony/datas/2019-05-01/2019-05-01-input.txt /home/tony/datas/2019-05-01/Main 2402:f000:4:38:808::382 5
time sudo python patternMining.py --read /home/tony/datas/2019-05-01/Main/AS/AS16509.hex --write /home/tony/datas/2019-05-01/Main/AS/AS16509.hex.target.noexperience --budgets 10000 100000 1000000 10000000 --depth 4 --experience=False

mining /home/tony/datas/2019-05-01/Main/AS/AS16509.hex


real	59m30.542s
user	59m22.755s
sys	0m5.508s


一键化 预算 10000 100000 1000000 10000000
两种预测方式
sudo bash all.sh /home/tony/datas/2019-05-01/2019-05-01-input.txt /home/tony/datas/2019-05-01/Main 2402:f000:4:38:808::382

python patternMining.py --read /home/tony/datas/2019-05-01/Test/AS/AS13335.hex --write /home/tony/datas/2019-05-01/Test/AS/AS13335.target_a --budgets 1000 10000 100000 1000000 10000000 100000000 --depth 4

bash all.sh /home/tony/datas/2019-05-01/2019_h_10000.txt /home/tony/datas/2019-05-01/Test 2402:f000:4:38:808::382
sudo python AliveDetection.py --input=/home/tony/datas/2019-05-01/2019_h_10000.txt --filename=alive.txt --dir=/home/tony/datas/2019-05-01/Alive --IPv6=2402:f000:4:38:808::382

system
2019-05-01-input.txt
活跃地址探测
getAlive -> alive.txt
APD.py alive.txt
aliased.txt

aliases-lpm.py
non-aliased.txt

AS_split

for every file in directory:
	patternMining
-> predict files
	dedup
collect hit rate


def stat(lines, dic_):
	for line in lines:
		s = line.strip()
		if s=='':continue
		index=s.find('/')+1
		length=int(s[index:])
		if length not in dic_:
			dic_[length] = 1
		else:
			dic_[length] += 1
sudo zmap --ipv6-target-file=2019-05-01-input.txt --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_alive.txt
存活率 20.26%

对AS6057作探测
cat AS6057.txt | ipv6-addr2hex > AS6057.hex 获得hex

进行预测
python patternMining.py --read /home/tony/datas/2019-05-01/AS/AS6057.hex --write /home/tony/datas/2019-05-01/AS/6057_target.hex --budgets 1000 10000 100000 1000000 10000000 100000000 --depth 4

由于数量过大,略
python Deduplication.py -A /home/tony/datas/2019-05-01/AS/6057_target.hex.100000000 -B /home/tony/datas/2019-05-01/AS/AS6057.hex -C /home/tony/datas/2019-05-01/AS/6057_dedup.hex.100000000
cat 9146_dedup.hex.10000 | ipv6-hex2addr > 9146_target.txt.10000
sudo zmap --ipv6-target-file=9146_target.txt.10000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.10000

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/6057_target.hex.10000000 -B /home/tony/datas/2019-05-01/AS/AS6057.hex -C /home/tony/datas/2019-05-01/AS/6057_dedup.hex.10000000
cat 6057_dedup.hex.10000000 | ipv6-hex2addr > 6057_dedup.txt.10000000
sudo zmap --ipv6-target-file=6057_dedup.txt.10000000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 6057_result.txt.10000000
0.07%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/6057_target.hex.1000000 -B /home/tony/datas/2019-05-01/AS/AS6057.hex -C /home/tony/datas/2019-05-01/AS/6057_dedup.hex.1000000
cat 6057_dedup.hex.1000000 | ipv6-hex2addr > 6057_dedup.txt.1000000
sudo zmap --ipv6-target-file=6057_dedup.txt.1000000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 6057_result.txt.1000000
0.10%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/6057_target.hex.100000 -B /home/tony/datas/2019-05-01/AS/AS6057.hex -C /home/tony/datas/2019-05-01/AS/6057_dedup.hex.100000
cat 6057_dedup.hex.100000 | ipv6-hex2addr > 6057_dedup.txt.100000
sudo zmap --ipv6-target-file=6057_dedup.txt.100000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 6057_result.txt.100000
0.13%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/6057_target.hex.10000 -B /home/tony/datas/2019-05-01/AS/AS6057.hex -C /home/tony/datas/2019-05-01/AS/6057_dedup.hex.10000
cat 6057_dedup.hex.10000 | ipv6-hex2addr > 6057_dedup.txt.10000
sudo zmap --ipv6-target-file=6057_dedup.txt.10000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 6057_result.txt.10000
0.13%

wc -l 6057_dedup.txt.10000 6057_dedup.txt.100000 6057_dedup.txt.1000000 6057_dedup.txt.10000000
    33286 6057_dedup.txt.10000
    67294 6057_dedup.txt.100000
   602848 6057_dedup.txt.1000000
  6734152 6057_dedup.txt.10000000
  7437580 total
wc -l 6057_result.txt.10000 6057_result.txt.100000 6057_result.txt.1000000 6057_result.txt.10000000
    43 6057_result.txt.10000
    89 6057_result.txt.100000
   621 6057_result.txt.1000000
  4565 6057_result.txt.10000000
  5318 total

================
wc -l 9146_target.txt.10000 9146_target.txt.100000 9146_target.txt.1000000 9146_target.txt.10000000 9146_target.txt.100000000
      2332 9146_target.txt.10000
     29484 9146_target.txt.100000
    424172 9146_target.txt.1000000
   6524222 9146_target.txt.10000000
  91411007 9146_target.txt.100000000
  98391217 total
wc -l 9146_result.txt.10000 9146_result.txt.100000 9146_result.txt.1000000 9146_result.txt.10000000 9146_result.txt.100000000
      2 9146_result.txt.10000
     27 9146_result.txt.100000
    684 9146_result.txt.1000000
   7805 9146_result.txt.10000000
  24124 9146_result.txt.100000000
  32617 total
预测地址标准化
python Deduplication.py -A /home/tony/datas/2019-05-01/AS/9146_target.hex.10000 -B /home/tony/datas/2019-05-01/AS/AS9146.hex -C /home/tony/datas/2019-05-01/AS/9146_dedup.hex.10000
cat 9146_dedup.hex.10000 | ipv6-hex2addr > 9146_target.txt.10000
sudo zmap --ipv6-target-file=9146_target.txt.10000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.10000
0.08%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/9146_target.hex.100000 -B /home/tony/datas/2019-05-01/AS/AS9146.hex -C /home/tony/datas/2019-05-01/AS/9146_dedup.hex.100000
cat 9146_dedup.hex.100000 | ipv6-hex2addr > 9146_target.txt.100000
sudo zmap --ipv6-target-file=9146_target.txt.100000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.100000
0.09%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/9146_target.hex.1000000 -B /home/tony/datas/2019-05-01/AS/AS9146.hex -C /home/tony/datas/2019-05-01/AS/9146_dedup.hex.1000000
cat 9146_dedup.hex.1000000 | ipv6-hex2addr > 9146_target.txt.1000000
sudo zmap --ipv6-target-file=9146_target.txt.1000000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.1000000
0.16%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/9146_target.hex.10000000 -B /home/tony/datas/2019-05-01/AS/AS9146.hex -C /home/tony/datas/2019-05-01/AS/9146_dedup.hex.10000000
cat 9146_dedup.hex.10000000 | ipv6-hex2addr > 9146_target.txt.10000000
sudo zmap --ipv6-target-file=9146_target.txt.10000000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.10000000
0.12%

python Deduplication.py -A /home/tony/datas/2019-05-01/AS/9146_target.hex.100000000 -B /home/tony/datas/2019-05-01/AS/AS9146.hex -C /home/tony/datas/2019-05-01/AS/9146_dedup.hex.100000000
cat 9146_dedup.hex.100000000 | ipv6-hex2addr > 9146_target.txt.100000000
sudo zmap --ipv6-target-file=9146_target.txt.100000000 --ipv6-source-ip=2402:f000:4:38:808::382 -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o 9146_result.txt.100000000
0.03%

cat AS9146.txt | ipv6-addr2hex > AS9146.hex
python patternMining.py --read /home/tony/datas/2019-05-01/AS/AS9146.hex --write /home/tony/datas/2019-05-01/AS/9146_target.hex --budgets 1000 10000 100000 1000000 10000000 100000000 --depth 4
python patternMining.py --read /home/tony/datas/2019-05-01/AS/AS9146.hex --write /home/tony/datas/2019-05-01/AS/9146_target_10w.hex --budget 100000 --depth 2
python AS_split.py --src /home/tony/datas/2019-05-01/2019-05-01-non-aliased.txt --dst /home/tony/datas/2019-05-01/AS --dat /home/tony/Desktop/ipasn_20190402.dat
python AS_split.py -analyze --min 0 --dst /home/tony/datas/2019-05-01/AS
python AS_split.py -split --dst /home/tony/datas/2019-05-01/AS --mode AS --value 10
统计前缀
1145702 aliased
253763462 non-aliased
254909164
python aliases-lpm.py -a /home/tony/datas/2019-05-01/aliased_prefixes.txt -i /home/tony/datas/2019-05-01/2019-05-01-input.txt --non-aliased-result /home/tony/datas/2019-05-01/2019-05-01-non-aliased.txt --aliased-result /home/tony/datas/2019-05-01/2019-05-01-aliased.txt

254000000 1850.83821201 seconds
python APD.py -extract -i /home/tony/datas/2019-05-01/2019-05-01-input.txt -db /home/tony/datas/2019-05-01/APD.db
python APD.py --detectOne --database /home/tony/datas/2019-05-01/APD.db --threshold 100 --IPv6 2402:f000:4:38:808::382 --directory /home/tony/datas/2019-05-01/tempdir
python APD.py -output --database /home/tony/datas/2019-05-01/APD.db > /home/tony/datas/2019-05-01/aliased_prefixes.txt

python APD.py -extract -i /home/tony/datas/2019-05-01/2019_h_10000.txt -db /home/tony/datas/2019-05-01/test.db
python APD.py --detectOne --database /home/tony/datas/2019-05-01/test.db --threshold 2 --IPv6 2402:f000:4:38:808::382 --directory /home/tony/datas/2019-05-01/smalltest

时间
78800000 660.336174965
254000000 1783.37809205
254000000 1819.1710639 seconds
indexing use 1193.42084002 seconds
254909164 use 462.430617809 seconds
254909164 use 468.080968857 seconds
254909164 use 1260.32825899 seconds
为05-01去别名
python APD.py -extract -i /home/tony/datas/2019-05-01/2019-05-01-input.txt -db /home/tony/datas/2019-05-01/APD.db -t 1000 > /home/tony/datas/2019-05-01/prefix_above_1000.txt
python APD.py -extract -i /home/tony/datas/2019-05-01/2019-05-01-input.txt -db /home/tony/datas/2019-05-01/APD.db -t 1000

python APD.py --test -db /home/tony/datas/2019-05-01/APD.db
==============
python AS_split.py -s ~/datas/2019-05-01/2019-05-01-input.txt -d ~/datas/2019-05-01/database --dat ~/Desktop/ipasn_20190402.dat
python AS_split.py -analyze -d ~/datas/2019-05-01/database
python AS_split.py -split -d ~/datas/2019-05-01/database
去重
python Deduplication.py -A results/9146_target_.hex -B ~/datas/2019-04-12/AS_result/9146_REST.hex -C results/9146.dedup.hex

python Deduplication.py -A results/9146_5-1_.hex -B ~/datas/2019-05-01/database/9146.tail.1M.hex -C results/9146_5-1.dedup.hex
转addr
cat results/9146.dedup.hex | ipv6-hex2addr > results/9146.scanTarget.txt
cat results/9146_5-1.dedup.hex | ipv6-hex2addr > results/9146_5-1.scanTarget.txt
扫描
sudo zmap --ipv6-target-file=results/9146_5-1.scanTarget.txt --ipv6-source-ip=2402:f000:9:8401:e4fd:ef33:c041:136e -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o results/9146.5-1_scanResult.txt
sudo zmap --ipv6-target-file=results/9146.scanTarget.txt --ipv6-source-ip=2402:f000:9:8401:e4fd:ef33:c041:136e -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o results/9146.scanResult.txt
sudo zmap --ipv6-target-file=results/9146.scanTarget.txt --ipv6-source-ip=2402:f000:9:8401:e4fd:ef33:c041:136e -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o results/9146.scanResult.txt
测试是否存活
cat ~/datas/2019-04-12/AS_result/9146_REST.hex | ipv6-hex2addr > results/9146.REST.txt
sudo zmap --ipv6-target-file=results/9146.REST.txt --ipv6-source-ip=2402:f000:9:8401:e4fd:ef33:c041:136e -M icmp6_echoscan -f saddr,daddr,ipid,ttl,timestamp_str -O json -o results/9146.REST.alive.txt
model

列表A,B
循环
完全展开法:
展开直到:
	每一项都
	maxN==1
避免排序:
	保证maxN=1的连续

算法:
列表A, B
初始化:A=[[*, IPs]], B=[[item,density,range]]
循环:
	range=calculate_range(B)
	if range <= budget:
		goto end
	x=B[0]
	newitem = expand(x)
	A,B.append(newitem)
	A,B.remove(x)
	
优化
显著的pattern识别:
减少循环次数,提升预测效果
算法无需改变,初始化改变


总共32位
1。消除确定位 剩余32~16
2。统计4位的
	最坏情况,32位都是随机的,此时找出密度最大的区域

3。

找出现最多的一位,重复16次(或者限定百分比)
扫描缩小范围
比较范围和预算
优先缩小密度小的pattern

eui and lowbytes:
6057.hex 71M		2.4G not able to process in memory
9146.hex 25M		871M able to process in memory
6057.hex  98% eui-64
9146.hex  50% eui-64
6057.hex  500s 0.1M lowbytes 0.1%
9146.hex	211s 10M lowbytes 40%

========================
EUI-64 dealer
n*2**24
can n be small?
24|16|24
to statistic


别名前缀--------多长的合适

latest input if too much use part of it
* find aliased prefixes , compare with the website
* get rid of aliased addresses
* show ASN distribution
* 

是否是别名地址|属于哪个AS|属于哪些pattern|
patter的不确定位的特征挖掘
1.kmeans
2.DBSCAN
3.Natural Breaks

=============
前缀树
一个多叉树,叉数最多为16
挑选多叉数的节点,组成表征集合,目的是表现分布
如果父节点属于集合,则子节点不属于集合,
越少的前缀数量,越多的包含数量
过短的前缀? 同时密度越大越好-即前缀越长越好
密度大的都是前缀15的
pattern占比
前缀长度15的个数很多,因此一项的密度较小
要求 100个前缀 覆盖 90%的地址 最大密度
不同长度前缀取前100个,覆盖率>90%

=======================
pattern=xee77xxxxxxxxxxx, match count=200938, percentage=0.0673515178208
pattern=aee7xxxxxxxxxxxx, match count=2, percentage=6.70371137573e-07
pattern=xxe77bxxxxxxxxxx, match count=1, percentage=3.35185568786e-07
任务,聚合成一类
xee77xxxxxxxxxxx
==========
APD
clusters 
/64 -> /124  /120 (地址空间大小2**8=256)
100 targets
假设检测的前缀长度为l
其对应的地址空间为2**(128-l)
/64-/96

输入:IPv6地址集
统计所有长度在[min,max]之间的前缀的数量
最大15倍输入的数量
过滤少于n~100的前缀
对于剩余的前缀
生成随机地址
探测
统计相应率和别名前缀
生成别名前缀列表

去别名
=====================
问题:
python内存占用过大
任务:
每个IP40字节
共102M个IP   2M-7s   350s
建索引 100s


					-1 					63949
NO			63 17 13			46 13 4	
INDEX		51 11 14   		47 1.7 3.4
每个IP 用int存 = 24B
共 2.4G内存?

用sqlite3数据库!
create table IPs(address,AS_number)
conn.executemany('insert into IPs values (?,?)',[('aaaaaaaa',102),('aaaaaaab',103)])
conn.execute('select * from IPs')
conn.execute('select distinct AS_number from IPs')
connection.execute('select count(*) from IPs;')
connection.execute('create index AS_index on IPs (AS_number)')
b=connection.execute('select address from IPs where AS_number=-1')
b=connection.execute('select address from IPs INDEXED BY AS_INDEX where AS_number=-1')
======================
总体流程
APD
AS
Pattern
Predict
Validate
===================

APD 模块设计
======================
ASN 模块设计
输入:IPv6数据集+目标保存文件夹
两个string
输出:所有AS号和对应的前缀列表
输出2:每个AS号对应的IP集合
输出3:每个AS号对应的IP的数量&比例 可视化
=====================================
28	print(IP[:16]+'|'+IP[16:])
cluster.txt.27 画 percentage-pattern.jpg
cluster.txt.26 画出每个pattern最好的折线图
cluster.txt.25 消除包含关系
cluster.txt.24 用于获取测试数据
进行计数器的定义和实现

cluster.txt.17 >后对最后一位不同的pattern进行了聚合
cluster.txt.20 >=后对连续c位中的确定位进行了消除 205->166
1.可视化pattern
2.预测

entropy-ip
预测方法:
cpd 条件概率密度
从左到右,
根据已经有的值 和cpd的概率 进行随机

eip-generator

一个地址由n个段组成
每个段有多个取值范围和贝叶斯概率

建立模型,计算出所有段的所有组合的最终概率和对应的地址数量

若概率>1/N,则 对该模型输出 min(p * N, M)个地址 这些地址都是在范围内独立随机的

假如模型有4M个,每个模型对应的概率均匀,=1/4M

生成 1M 个地址,由于概率太小不输出地址



在单个网络内使用的方法不涉及熵
在多个网络之间使用熵聚类
熵IP的熵似乎只用来作图了
熵聚类的熵用来聚类网络


entropy-ip:
1.计算32个hex的熵
2.根据规则划分段
3.用bnfinder计算贝叶斯网络



entropy-ip预测时只考虑出现过的IP地址的组合?
1.是:加入密集区域
没有用聚类结果

2.DBSCAN
在a2-mining寻找高密度区域时使用
outliners 多次出现的
highly dense ranges 
uniformly distributed ranges 


识别出随机的地址——代价大的地址
脚本entropy-calculate.py
输入 文件名
输出 熵排序后的地址
可视化 每个地址的熵



tool
  input: IPv6 address
  step1:
    extract n address randomly
  step2:
    split n address into K fold
    gen K sub directory
    one train , one test
  step3:
    run specific model on data

step1 and step2 can be universal  数据集抽样和K折生成

step3 is different

tool2 数据集分类
  input: IPv6 address
  function:
     to classify into different types
  types:
     Low Byte
	1.ffffff
		2^24
	2.00ffff
		2^16
	3.ff00ff
		2^16
	4.0000ff
		2^8
	
     IPv4 embedding
	1.00ff00ff00ff00ff
	2.00000000ffffffff
	2^33 too large
	except 0.x.x.x i.e. 00xxxxxx
     SLAAC
		 fffe
		2^48
     Port
	1.ff0080
	 .ff0050
	2.8000ff
	 .5000ff
	for port n:
        2^8*4=2^10
	20 popular port:
	[20,21,22,23,25,53,67,68,69,80,110,123,137,138,139,143,161,162,179,389,443,636,989,990]
	

tool3 
	数据集前缀分析
	prefix.txt 前缀和数量 比例
问题
  相同前缀的IPv4地址比例较小
	
	前缀别名分析

目标
	1.识别单词
	face
	cafe
  2.识别faceb00c
	3.识别网络前缀
		step1:
		  识别数据里64前缀个数n
		step2:
			根据前缀划分数据集n
			在每个数据集上计算某特征
			a.单词
			b.分配比例
			c.
		step3:
连续的匹配
输出:字典
/64前缀 发现的字典
{
	'/64':{'word1':a1,'word2':a2}
}
IID
每4个hex记录

选取密度最大的前缀
进行低字节扫描

数量控制在1w左右
10个前缀,
只扫描000fff 2^12=4k


为不同的路由前缀分配不同的预算可能是很自然的.
例如,路由前缀的预算可能取决于其内种子的数量或前缀本身的大小。但是,这可能会严重扭曲目标生成向更密集的网络方向发展,以牺牲找到的活跃地址数量的多样性。最合适的预算分配政策是什么,以及这与目标生成目标的区别,仍然是一个悬而未决的问题。


1.别名识别
2.前缀、子网 划分
3.熵聚类 
4.6gen

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages