Repository navigation
Expand file tree
/
Copy pathsomeHackDaysNotes.txt
More file actions
128 lines (83 loc) · 10.5 KB
/
Copy pathsomeHackDaysNotes.txt
File metadata and controls
128 lines (83 loc) · 10.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
similar
suggestions
bought together
front end
where? possible to run everything on my dude??????? yes that would be best. build network on fancy aws machine but host elasticsearch and website and and predictions and rest endpionts on dude.
ehhhhhhhhhh dsstne needs gpus
elasticsearch - loaded with product titles and descriptions and skus and image urls per siteID
where? aws? my dude? no don't waste time on typeahead, suggest
schema:
product id search EXACT (type: keyword)
image url ------------- no search (elasticsearch: set index to false https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index.html)
product title search
product description search
rest endpoints:
1) search elasticsearch for product - get dropdown and typeahead?
where? this can be anywhere
2) return product suggestions
where? machine running model. with neural network in memory. in redis? or maybe disk is fine if aws solid state? no, loading file took longest time.
machine needs to access elasticsearch.
or just use java ... keep ..... wait no ... the thing is in C. just needs to interact with command line of machine. generate recommendations file. read file.
rest machine should:
generate recommendations file -- use command line
calculate recommendations -- use command line
read recommendations result -- use command line
possibly search elasticsearch for similar products as recommended
...lets just use java. can C read from redis? how?
IMAGE DOWNLOADER
- how did i do this in pxje ... whatever, don't use imagemagick. just use java solution. okay if images not perfect.
- crap this depends on how many images we need. and how many products contain image urls
NOTE - don't download images! just save the bing search results thumbnail: https://tse4.mm.bing.net/th?id=OIP.x-ER3WkOYbzDFlCxo-SD9wEsC7&w=297&h=185&c=7&qlt=90&o=4&dpr=2&pid=1.7
TODO - elasticsearch, rest endpoints (runs predictions),
0. do dssnte example again on dummy data but save the model file. try using it on a smaller amazon vm.
1. build vue website and serve from dude
2. build java endpoints and serve from dude
3. install elasticsearch on dude
4. hook up java Dude endpoint to query elasticsearch
5.
java projects
- image downloader - how?????? pay for google images? use bing. mediocre images okay.
- non dsstne rest endpoints
- dsstne rest endpoints
<img class="mimg" style="color: rgb(199, 178, 4);" height="189" width="170" src="https://tse3.mm.bing.net/th?id=OIP.d2xT0PAAQf5lj57_i8ZG_wENEs&w=170&h=189&c=7&qlt=90&o=4&dpr=2&pid=1.7" alt="Image result for pesticide" data-bm="22">
okay... dummy endpoint projects are up. elasticsearch is installed. image downloader project is up.
what next ...flush out endpoints. write specific endpoints:
GPU:
return product ES documents.
dssnte-related tools. get product ids, determine product ids. use dude apis to get documents and return those documents
Dude:
1. get product IDs, return the product Documents
2. BoughtTogether - get product ids, return bought together product Ids. products info is loaded in memory in java? yep
TODO -
X -1. add projects to same repo
X 0. create dummy data
X 1. load some dummy data in elasticsearch. (products (title, id, desc, image)). review elasticsearch schema. - do this now. first set schema
2. build dude endpoints to: (1) query elasticsearch (get prod document(s) for input id(s)), (2) calculate boughtTogether
3. DO DSSTNE STUFF WTIH DUMMY DATA
build RNN model w/ dummy data (add more products to dummy data first tho)
copy network model file to repo or S3
write code to deploy smaller gpu instance on-demand
write code to auto-terminate gpu instance after a few hours of inactivity.
build java endpoint to accept products and run RNN model for suggestions
4. build vue SPA. find tutorial. use routers.
5. look into NLP model for "look at this next" suggestion
6. move hackdays projects to same github repo
next word prediction w/ tensorflow
https://stackoverflow.com/questions/33773661/predicting-next-word-using-the-language-model-tensorflow-example
https://arxiv.org/pdf/1602.06291.pdf
https://rpubs.com/pcbrom/nextwordpres
https://github.com/bburns/LanguageModels/blob/master/docs/report/report.pdf
https://medium.com/towards-data-science/lstm-by-example-using-tensorflow-feb0c1968537
http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
build dummy data.
how? make several groups of products. (
cookout,
home maintenance (nails, caulking, hammer, screws, wd40),
movie party,
gardening,
star wars toys,
domestic disposables (TP, PT, hand soap, detergent, laundry soap)
make a bunch customers. for each customer, assign them ONE of the groups ^ and select a number of products for them. make MOST of their products from the same group. make 10% of products orders, the rest browse.
what code? lets use dude endpoint. no, use image downloader, cos it's like utility thing that puts stuff in elasticsearch.