Create embulk config subcommand

[takkanm]

ref #80

I implemented connector:init that creates a configuration file of 'connector:guess' from options similar to 'import:prepare', 'import:upload'.

Now, supoprt format is S3 and mysql.

$ td connector:init "s3://access_key:secret_key@/my_bucket/path/to/*.csv" -o seed.yml
initialize configuration:

---
in:
  type: s3
  access_key_id: access_key
  secret_access_key: secret_key
  bucket: my_bucket
  path_prefix: path/to/
out:
  mode: append

Created seed.yml file.
If you need to change, please modify seed.yml.
Use 'td connector:guess seed.yml' to create bulk load configuration.

$ td connector:init mytable --format mysql --db-url jdbc:mysql://localhost/mydb --db-user myuser --db-password mypass
initialize configuration:

---
in:
  type: mysql
  host: localhost
  port: 3306
  database: mydb
  user: myuser
  password: mypass
  table: mytable
  select: "*"
out:
  mode: append

Created td-bulkload.yml file.
If you need to change, please modify td-bulkload.yml.
Use 'td connector:guess td-bulkload.yml' to create bulk load configuration.

[takkanm]

@nahi @muga pleaze, review it.

[frsyuki]

I think subcommand should not be in connector: group because it's a migration tool for legacy a tool where new users don't have to be aware of it.
Why don't you add it to import: group? For example (I don't say this is the best):

$ td import:config logs/*.csv --format csv --columns time,uid,price,count --time-column time -o parts/
Generating seed.yml...
    Using local file input
    Using CSV parser plugin
    Using Treasure Data output
Done. Please use embulk to load the files.
Next steps:

    # install embulk
    $ embulk gem install embulk-output-td
    $ embulk guess seed.yml -o config.yml
    $ embulk preview config.yml
    $ embulk run config.yml

$ td import:config "s3://<s3_access_key>:<s3_secret_key>@/my_bucket/path/to/*.csv" --format csv --column-header --time-column date_time -o parts/
Generating seed.yml...
    Using S3 input
    Using CSV parser plugin
    Using Treasure Data data connector
Done. Please use connector:guess and connector:run to load the files.
Next steps:

    $ td connector:guess seed.yml -o config.yml
    $ td connector:preview config.yml
    $ td connector:run config.yml

As you notice, import from local files and most of mysql servers are not supported by data connector. In this case, we need to use embulk + embulk-output-td.

@nahi @muga any thoughts?

[muga]

I think subcommand should not be in connector: group because it's a migration tool for legacy a tool where new users don't have to be aware of it.

Agree. it's better that this feature should be moved to import: group.

As you notice, import from local files and most of mysql servers are not supported by data connector. In this case, we need to use embulk + embulk-output-td.

Nice. Since most S3 buckets can be accessed by data connector, it can be migrated to data connector.

• local files -> embulk + embulk-output-td
• mysql -> embulk + embulk-output-td
• s3 files -> data connector

[takkanm]

Thank you for reviewing.
I think that it is trying to respond as follows.

• command name is import:config
• s3 files is data connector
• local files and mysql is embulk + embulk-output-td

[toru-takahashi]

import:config looks good.

Also, please check the following cases.

1. If user don't have time column, they uses --time-value TIME,HOURS option.
2. If user uses --column-header and --columns option. It should be used --columns name without reading header line.

[muga]

@toru-takahashi Right. That would help users' migration.

[takkanm]

I have one question. Should td import:config support --column option ?

td import:config create configuration file for guess commands.
I am understanding that guess commands create new column option from source file.
Therefore, I will leave it to guess about the column, I think that it is good for the user to edit.
How about you ?

[muga]

@takkanm That's right.

I thought that columns option should be migrated to rename filter's config. But the config requires existing column names. In seed.yml, existing column names are not defined.

column-header option could be ignored because, you know, import:guess command just generates seed.yml only.

[takkanm]

@muga Thank you. So, I think I try to issue a warning message if user set --column and --column-header option.
How about you ?

[muga]

@takkanm sounds perfect. please go ahead.