nwztools/database: misc improvements

* make gen_db.py work on Windows/Python 2

- use hashlib module instead of md5sum, also don't rely on / for file path
matching
- don't use 'file' for a variable name

* fix parse_nvp_header.sh for older kernels

pre-emmc kernel sources use a slightly different #define format; adjust
regexp to catch it.

* add nwz-x1000 series NVP layout (from icx1087_nvp.h)

some new tags have no description, alas the driver doesn't have
them :/

*  minor fixes to nvp/README

fixed typos/wording

Change-Id: I77d8c2704be2f2316e32aadcfd362df7102360d4

4 files changed, 111 insertions, 22 deletions

diff --git a/utils/nwztools/database/nvp/README b/utils/nwztools/database/nvp/README
index cb403291c8..5ff18513d7 100644
--- a/utils/nwztools/database/nvp/README
+++ b/utils/nwztools/database/nvp/README
@@ -1,38 +1,39 @@
 The NVP map varies a lot from players to players, it is inconceivable to build
 it by hand. The approach taken is to extract it from the kernel of each player.
 Since Sony provides the kernel of all players, it is 'only' a matter of
-downloading all of them. A bit of back story on the NVP: it is non-volatile
-area of the flash that is divided in regions and then "sectors" (unrelated to
-hard drive sectors). Each "sector" stores the data of a "node". The ABI
+downloading all of them. A bit of background on the NVP: it is non-volatile 
+partition of the flash that is divided in regions and then "zones". 
+Each "zone" stores the data of a "node". The ABI
 between the NVP driver and the userspace is an index: the userspace gives the
-index of a node, and then drives looksup its table to see where it is and what
+index of a node, and then drives looks up its table to see where it is and what
 is its size. The index map changes over time so Sony introduces standard "names"
 for its entries, those are 3-letters acronym (for example "fup" or "bti" or "shp")
-that have a corresponding index. For some reason, the driver also contains a
-description of the nodes, in english (so "bti" stands for "boot image").
+that have a corresponding index. Sometimes the driver also contains a
+description of the nodes, in English (e.g. "bti" stands for "boot image").
 
 parse_nvp_header.sh
 ===================
 
-This script is given a file name, a kernel directory or a kernel tgz and it will
+This script takes a header filename, a kernel directory or a kernel tgz and will
 try to extract the mapping automatically. It produces a list of pairs
   <node>,<name>
 where <node> is the index of the node (that's the only thing that is usable on
-a running device) and <name> is the standard name of the node. Note that is
-some kind of acronym (like 'fup') and the description needs to be generated
-separatly (see other section).
+a running device) and <name> is the standard name of the node. The output should
+be written to <series name>.txt
+Note that <name> is an acronym (like 'fup') and the description needs to be generated
+separately (see nvprool section).
 
 parse_all_nvp_headers.sh
 ========================
 
-This scripts expects a directory to have the following structure:
+This script expects a directory to have the following structure:
   dir/
     nwz-a10/
       linux-kernel-*.tgz
     nwz-e460/
-      linxu-kernel-*.tgz
+      linux-kernel-*.tgz
     ...
-Each sudirectory must the series name (as used in ../series.txt) and the kernel
+Each sudirectory must be the series name (as used in ../series.txt) and the kernel
 must be a tgz (end in .tgz and not .tar.gz) of the form linux-kernel-*.tgz. Usually
 the variable bit will be the version but some kernels have unknown versions. It
 will then run parse_nvp_header.sh on each of them and store the result in a
@@ -43,24 +44,24 @@ NOTE: the kernel can be symlinks to other files
 nvptool
 =======
 
-The kernel headers do no contain the description of the nvp node names.
-This one can be extract from the icx_nvp[_emmc].ko driver on target using complicated
-elf parsing done by nvptool. Technically nvptoo can find much more information
-like the node -> humanname mapping as well and the actual sector on the disk but
+The kernel headers do not contain descriptions of the nvp node names.
+They can be extracted from the icx_nvp[_emmc].ko driver on target using complicated
+elf parsing done by nvptool. Technically nvptool can discover much more information
+like the node -> human name mapping as well and the actual zone in the flash but
 since we can already extract it easily from the headers, we only extract description
 names from it.
 
 parse_all_nvp_nodes.sh
 ======================
 
-This scripts expects a directory to have the following structure:
+This script expects a directory to have the following structure:
   dir/
     nwz-a10/
       rootfs.tgz
     nwz-e460/
       rootfs.tgz
     ...
-Each sudirectory must the series name (as used in ../series.txt) and the rootfs
+Each sudirectory must be the series name (as used in ../series.txt) and the rootfs
 must be a tar. It will then extract the relevant icx_nvp driver from it and run
 nvptool on it to produce a file called nodes-<series name>.txt
 
diff --git a/utils/nwztools/database/nvp/nwz-x1000.txt b/utils/nwztools/database/nvp/nwz-x1000.txt
new file mode 100644
index 0000000000..ab0f7fb872
--- /dev/null
+++ b/utils/nwztools/database/nvp/nwz-x1000.txt
@@ -0,0 +1,82 @@
+app,0
+bti,1
+hdi,2
+cng,3
+ser,4
+dg0,5
+dg1,6
+dcc,7
+mdl,8
+fup,9
+bok,10
+shp,11
+dba,12
+dbv,13
+tr0,14
+tr1,15
+mid,16
+tst,17
+gty,18
+fui,19
+lbi,20
+dor,21
+edw,22
+ubp,23
+syi,24
+var,25
+pcd,26
+dbs,27
+rnd,28
+ufn,29
+sdp,30
+ncp,31
+kas,32
+pnc,33
+rtc,34
+bpr,35
+e00,36
+e01,37
+e02,38
+e03,39
+e04,40
+e05,41
+e06,42
+e07,43
+e08,44
+e09,45
+e10,46
+e11,47
+e12,48
+e13,49
+e14,50
+e15,51
+e16,52
+e17,53
+e18,54
+e19,55
+e20,56
+e21,57
+e22,58
+e23,59
+e24,60
+e25,61
+e26,62
+e27,63
+e28,64
+e29,65
+e30,66
+e31,67
+clv,68
+slp,69
+ipt,70
+ep0,71
+ep1,72
+ep2,73
+ep3,74
+pts,75
+skt,76
+mac,77
+apd,78
+blf,79
+hld,80
+skd,81
diff --git a/utils/nwztools/database/nvp/parse_nvp_header.sh b/utils/nwztools/database/nvp/parse_nvp_header.sh
index 8baab3c9eb..ee2be93516 100755
--- a/utils/nwztools/database/nvp/parse_nvp_header.sh
+++ b/utils/nwztools/database/nvp/parse_nvp_header.sh
@@ -66,13 +66,16 @@ else
     >&2 echo "Analyzing $FILE"
 fi
 
+# old format: #define ICX1087_NVP_NODE_APP "/dev/icx1087_nvp/0"
+# new format: #define ICX_NVP_NODE_APP ICX_NVP_NODE_BASE "0"
+
 cat "$FILE" | awk ' \
 BEGIN { \
-    expr = "#define[[:space:]]+ICX_NVP_NODE_([[:alnum:]]+)[[:space:]]+ICX_NVP_NODE_BASE[[:space:]]*\"([[:digit:]]+)\""; \
+    expr = "#define[[:space:]]+ICX[[:digit:]]*_NVP_NODE_([[:alnum:]]+)[[:space:]]+(ICX_NVP_NODE_BASE[[:space:]]*\"|\"/dev.*_nvp/)([[:digit:]]+)\"";
 } \
 { \
     if($0 ~ expr) \
     { \
-        print(tolower(gensub(expr, "\\1,\\2", "g", $0)));
+        print(tolower(gensub(expr, "\\1,\\3", "g", $0)));
     } \
 }'
diff --git a/utils/nwztools/database/nvp/parse_nvp_nodes.sh b/utils/nwztools/database/nvp/parse_nvp_nodes.sh
index 456a707e5d..8fc4367a8f 100755
--- a/utils/nwztools/database/nvp/parse_nvp_nodes.sh
+++ b/utils/nwztools/database/nvp/parse_nvp_nodes.sh
@@ -4,6 +4,9 @@
 # parse_nodes.sh /path/to/rootfs/dir output_file
 # parse_nodes.sh /path/to/rootfs.tgz output_file
 #
+if [ -z "$NVP_LOG" ]; then
+    NVP_LOG=/dev/null
+fi
 if [ "$#" -lt 2 ]; then
     >&2 echo "usage: parse_header.sh /path/to/icx_nvp.ko|/path/to/rootfs/dir|/path/to/rootfs.tgz output_file"
     exit 1
@@ -56,4 +59,4 @@ else
     >&2 echo "Analyzing $FILE"
 fi
 
-./nvptool -x "$FILE" -o "$OUTPUT" >/dev/null
+./nvptool -x "$FILE" -o "$OUTPUT" >"$NVP_LOG"