Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Refine the documentation of im2rec #12606

Merged
merged 5 commits into from
Sep 20, 2018
Merged

Refine the documentation of im2rec #12606

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a9d40c4
Merge branch 'master' of https://github.com/apache/incubator-mxnet
stu1130 Sep 18, 2018
429684a
Merge branch 'master' of https://github.com/apache/incubator-mxnet
stu1130 Sep 19, 2018
603b553
update the example data format and link to each others
stu1130 Sep 19, 2018
faf4994
fix wording
stu1130 Sep 19, 2018
540917d
delete the typo
stu1130 Sep 19, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions docs/faq/recordio.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,35 +6,39 @@ RecordIO implements a file format for a sequence of records. We recommend storin
* Packing data together allows continuous reading on the disk.
* RecordIO has a simple way to partition, simplifying distributed setting. We provide an example later.

We provide the [im2rec tool](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc) so you can create an Image RecordIO dataset by yourself. The following walkthrough shows you how.
We provide the [im2rec tool](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc) so you can create an Image RecordIO dataset by yourself. The following walkthrough shows you how. Note that there is python version of [im2rec tool](https://github.com/apache/incubator-mxnet/blob/master/tools/im2rec.py) and [example](https://mxnet.incubator.apache.org/tutorials/basic/data.html) using real-world data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We provide two tools for creating a RecordIO dataset.

  • im2rec.cc - implements the tool using the C++ API.
  • im2rec.py - implements the tool using the Python API.

Both provide the same output: a RecordIO dataset.
(Then take this mention and add it later for "Next Steps". I don't think you want them leaving this FAQ/tutorial quite yet.)
You may want to also review the example using real-world data with im2rec.py.


### Prerequisites

Download the data. You don't need to resize the images manually. You can use ```im2rec``` to resize them automatically. For details, see the "Extension: Using Multiple Labels for a Single Image," later in this topic.

### Step 1. Make an Image List File

* Note that the im2rec.py provide a param `--list` to generate the list for you but im2rec.cc doesn't support it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provides

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you, but


After you download the data, you need to make an image list file. The format is:

```
integer_image_index \t label_index \t path_to_image
```
Typically, the program takes the list of names of all of the images, shuffles them, then separates them into two lists: a training filename list and a testing filename list. Write the list in the right format.

This is an example file:

```bash
95099 464 n04467665_17283.JPEG
10025081 412 ILSVRC2010_val_00025082.JPEG
74181 789 n01915811_2739.JPEG
10035553 859 ILSVRC2010_val_00035554.JPEG
10048727 929 ILSVRC2010_val_00048728.JPEG
94028 924 n01980166_4956.JPEG
1080682 650 n11807979_571.JPEG
972457 633 n07723039_1627.JPEG
7534 11 n01630670_4486.JPEG
1191261 249 n12407079_5106.JPEG
95099 464.000000 n04467665_17283.JPEG
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the decimal point really required? this is a bit weird.

Copy link
Contributor Author

@stu1130 stu1130 Sep 20, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the data.lst file is generated by im2rec.py instead of doing it manually, the label will have those decimal point. I think it would be less confused for users?
And the reason why it uses floating point is that the label value could be generated by the regression, e.g. 68.6 kg for a human body weight.

10025081 412.000000 ILSVRC2010_val_00025082.JPEG
74181 789.000000 n01915811_2739.JPEG
10035553 859.000000 ILSVRC2010_val_00035554.JPEG
10048727 929.000000 ILSVRC2010_val_00048728.JPEG
94028 924.000000 n01980166_4956.JPEG
1080682 650.000000 n11807979_571.JPEG
972457 633.000000 n07723039_1627.JPEG
7534 11.000000 n01630670_4486.JPEG
1191261 249.000000 n12407079_5106.JPEG
```

### Step 2. Create the Binary File

To generate a binary image, use `im2rec` in the tool folder. `im2rec` takes the path of the `_image list file_` you generated, the `_root path_` of the images, and the `_output file path_` as input. This process usually takes several hours, so be patient.

Sample command:
Expand Down
2 changes: 2 additions & 0 deletions docs/tutorials/basic/data.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,8 @@ print(mx.recordio.unpack_img(s))
You can also convert raw images into *RecordIO* format using the ``im2rec.py`` utility script that is provided in the MXNet [src/tools](https://github.com/dmlc/mxnet/tree/master/tools) folder.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the repo link

An example of how to use the script for converting to *RecordIO* format is shown in the `Image IO` section below.

* Note that there is a C++ version of [im2rec](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc), please refer to [here](https://mxnet.incubator.apache.org/faq/recordio.html) for more information.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't link only "here". Provide the full description of what the link is going to.
Note that there is a C++ API implementation of im2rec. Refer to the RecordIO FAQ for more information.


## Image IO

In this section, we will learn how to preprocess and load image data in MXNet.
Expand Down