Easy contributable internationalization process with Sphinx (PyCon APAC 2015 in Taiwan)

Easy contributable internationalization process with Sphinx

Takayuki Shimizukawa

Sphinx co-maintainer

Sphinx-users.jp

1

Who am I

@shimizukawa

1. Sphinx co-maintainer

2. Sphinx-users.jp

3. PyCon JP committee

BePROUD co, ltd.2

http://pycon.jp/

3

Early Bird Registration: ２１０ tickets.

Speaker Call for Proposals: until July 15th.

4

170 tickets gone

Agenda

1. Sphinx introduction & Setup for i18n

2. A easy way and Non easy ways to translate

3. Sphinx i18n feature

4. Automated translation process with several services

5. Tips, tricks, traps

5

1. Sphinx introduction& Setup for i18n

6

What is Sphinx?

Sphinx is a documentation generator

Sphinx generates doc as several output formats from the reST text

markup

7


Sphinx

reSTreSTreStructuredTe

xt(reST)

reST Parser

HTML Builder

ePub Builder

LaTeX Builder texlive

HTMLtheme

Favorite Editor

Many docs are written by SphinxFor examples

Python libraries/tools:Python, Sphinx, Flask, Jinja2, Django, Pyramid, SQLAlchemy, Numpy, SciPy, scikit-learn, pandas, fabric, ansible, awscli, …

Non python libraries/tools:Chef, CakePHP(2.x), MathJax, Selenium, Varnish

8


Relation between Sphinx and Docutils

Sphinx based upon Docutils library

Docutils/Sphinx supports reStructuredText it called reST, it's an extensible markup(PEP287 / accepted at 2002)

9


Sphinx

reST Parser

HTML Builder

ePub Builder

LaTeX Builder

HTMLtheme

docutils

Comparing Docutils and Sphinx

Docutils Sphinx

1. Single page

2. Link to others with full name

3. Cross reference in a page

4. Writers: html, latex, man, xml, ...

5. Standard reST markup specs

1. Multiple Pages

2. toctree to connect all pages under single tree structure

3. Cross reference over each other pages

4. Additional writers: html(w/ themes), pdf(latex), texinfo, epub, …

5. Additional markup specs:autodoc directive and so on

10


SPHINX

docutils

HTML Builder HTML theme(Jinja2)

gettext Builder

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx i18n feature (built-in)

11


reST Parser(directive / role)

doctree(Intermediat

e)

reSTreSTreStructuredText(reST)

Sphinx internals and externals

12


SPHINX

docutils

Sp

hin

x拡

張d

irect

ive /

role

reSTreSTreStructuredText(reST)

Several formats

HTML Builder

EPUB Builder

LaTeX Builder

HTML theme(Jinja2)

code highlighter(Pygments)

man, texinfo, text, ... Builder

gettext Builder

XML, man, texinfo, text,winhelp, qthelp, ...

TeXLiveor else

*.pot

*.po

I18N

*.mo OmegaT

Pootle

Transifex

Translation Tools, Services

Favorite Editor

Sphinx Extension Builder

Custom Theme

epub3, docx, dash, ...

Sp

hin

x E

xte

nsi

on

dir

ect

ive /

role

reST Parser(directive / role)

doctree(Intermediat

e)

$ pip install sphinx

Support tools for translation

How to install Sphinx with i18n tools

To build a sphinx document

$ pip install sphinx-intl$ pip install transifex-client=0.8

13


$ git clone https://github.com/shimizukawa/deepthought.git$ cd deepthought/doc$ make html...Build finished. The HTML pages are in _build/html.

"make html" command generates html files into _build/html.

make html once

14


Files & setup conf.py$ tree /path/to/deepthought+- deep_thought| +- __init__.py| +- api.py| +- calc.py| +- utils.py+- doc| +- _build/| | +- html/ | +- _static/| +- _template/| +- conf.py| +- index.py| +- make.bat| +- Makefile+- setup.py

15


Document source

Document build output

Target library for doc1. ...2. 3. language = 'zh_tw'4. locale_dirs = ['locale']5.

doc/conf.py

2. Easy contributable internationalization process with Sphinx

16

What is internationalization?

17


I18N

What is easy contributable?

18


Not a easy way (example 1/3)

The manual are provided only in the HTML files

Rewriting reST files

19



The manual are generated fromreST files and

docstrings in the source files

Rewriting reST filesRewriting docstrins in the source files.

20



OK, we are engineers. We can use git!

Learn gitLearn GitHubgit clone to get the codeTranslation (Yes! This is what I want to

do!)git commitgit pullSometimes resolve conflictgit push

21


Not easy vs easy

Not a easy way Easy way

1. Learn git2. Learn GitHub3. git clone to get the

code4. Translation5. git commit, pull,

push6. Resolve conflict7. Build html your self

1. No git2. No Github3. No file4. Translation5. Update

Automatically6. No conflict7. You can get a HTML

output w/o hand-build.

22



23

Two i18n features of Sphinx

Output pot files:from reST

Input po files:to generate translated HTML

24


reST

pot

reST

html

po

Translation flow

Generate pot

Translate pot into po

Generate Translated HTML

25


reST

pot

reST

html

po

pot po

Translator

Pleasetranslate

Translate!Thanks for yourContribution!

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr ""

msgid "For example:"msgstr ""

26

Output pot files 3. Sphinx i18n feature

$ make gettext...Build finished. The message catalogs are in _build/gettext.

$ ls _build/gettextapi.pot examples.pot generated.pot index.pot

generated.pot

reST

pot

Preparing po files to translate

doc+- _build/| +- gettext/| +- api.pot| +- examples.pot| +- generated.pot| +- index.pot+- locale/

doc+- _build/+- locale/ +- zh_tw/ | +- LC_MESSAGES/ | +- api.po | +- examples.po | +- generated.po | +- index.po +- ja/

copy and renamefor each langs

Tra

nsla

te th

em

Translator

pot po

27


28

Preparing po files to translate

$ sphinx-intl update -p _build/gettext -l zh_twCreate: locale/zh_tw/LC_MESSAGES/api.poCreate: locale/zh_tw/LC_MESSAGES/examples.poCreate: locale/zh_tw/LC_MESSAGES/generated.poCreate: locale/zh_tw/LC_MESSAGES/index.po

At first, sphinx-intl copy pot files and rename them

pot po

sphinx-intl

$ make gettext$ sphinx-intl update -p _build/gettext -l zh_twNot Changed: locale/zh_tw/LC_MESSAGES/api.poUpdated: locale/zh_tw/LC_MESSAGES/examples.po +3, -1Not Changed: locale/zh_tw/LC_MESSAGES/generated.po

After change the document, sphinx-intl update differences


Translate po files

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr ""

generated.po

pot po

sphinx-intl

Translator

#: ../../../deep_thought/utils.py:docstring of deep_thought.utils.dumps:1msgid "Serialize `òbj`` to a JSON formatted :class:`str`."msgstr "msgstr " 序列化 `òbj`` 要 JSON 格式 :class:`str`."

generated.po

Translate by using Vim, Emacs, OmegaT, ...

29


Input po files

30


reST

html

po

Translated

$ make html...Build finished. The HTML pages are in _build/html.

Big picture

31


reST

pot

html

po

make gettext

sphinx-intl

Translator

make html

Doc Author

TranslationCatalog

Translatedcatalog


32

Entire process to translate sphinx docs

33


reST

pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

Translator

Translator

Translator

Autor / Translator

upload

Translator

clone

Translator

To Be

34


reST

pot

html

po

make gettext

sphinx-intl

make html

Doc Author

TranslationCatalog

Translatedcatalog

upload

Translators

To BeAutomated

To BeParallel

clone

Translation tool types

Vim / Emacs (Editors)Edit local filesTranslation support plugins are available.

OmegaT (Translation Tools)Edit local files.Translation support features.

Transifex (Services)Edit onlineTranslation support

features.

35


po

Translators

To BeParallel

Be Parallel by using Transifex

Transifex provides...As API:Upload potDownload po

As Web console:GlossaryTranslation memoryRecommendationAuto-translation

36


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-client

Download

Translation on Transifex web console

37


Original Text

Translated Text(you should keep reST syntax)

Suggestions fromTranslation Memory (TM)

Original(pot)

Translation(po)

Copy orig to translation

Machine translation

Save

Review (if needed)

Translators

Parallel

1

2

4

3

5

6

To Be Automated

38


po

Translators

Parallel

pot

Upload

pot

Auto Update

sphinx-intltransifex-client

po transifex-client

Download

reST

html

make gettext

make html

Doc Author

upload

To BeAutomated

clone

To Be Automated

39


pot

Upload sphinx-intltransifex-client

po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

The procedure for automation

1. clone repository2. make gettext3. Upload pot4. Download po5. make html6. Upload html

40


pot


po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomate

d

1. pip install sphinx sphinx-intl transifex-client==0.82. git clone https://github.com/shimizukawa/deepthought.git3. cd deepthought/doc4. sphinx-intl create-transifexrc # create ~/.transifexrc5. sphinx-intl create-txconfig # create .tx/config6. make gettext7. sphinx-intl -p _build/gettext update-txconfig-resources

# update .tx/config8. tx push -s # push pot files to

transifex9. tx pull -l zh_tw # pull po files from

transifex10. make html SPHINXOPTS="-D language=zh_tw"

Shell commands for automation

run.sh

$ export SPHINXINTL_TRANSIFEX_USERNAME=mice$ export SPHINXINTL_TRANSIFEX_PASSWORD=42$ export SPHINXINTL_TRANSIFEX_PROJECT_NAME=deepthought-0_7$ export SPHINXINTL_POT_DIR=_build/gettext$ run.sh

41


The drone.io

42

WebHook

Deploy

Clone repositoryRun shell script

The drone.io is a continuous integration service.


GitHub + drone.io + S3

43

GitHub

Amazon S3

Transifex 1. Clone repository2. make gettext.3. Upload pot4. Download po5. make html6. Upload html


2

1

3

Be Automated

44

pot


po transifex-client

Download

reST

html

make gettext

make htmlupload

clone1

2 3

456

To BeAutomated

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


Automated by drone.io

45

UploadpotDownload

poupload

WebHookclone

1

5

6make html

make gettext2

3

4

1WebHook

Automated


View from doc author

Doc Author doesn't require annoying procedure.

UpdateTranslation Source

Doc Author

Notify

See

Commit

46


View from doc translators

No gitNo fileNo conflictUpdate AutomaticallyThey can get a translated HTML output w/o hand-

build.

Translators

Parallel

See

TranslateTranslated Pages

47


The entire automated process

48

UpdateTranslation Source

Doc Author

Translators

Parallel

Notify

See

See

Publish

Translate

Commit

DownloadTranslations

Notify

Automated



49

TIP: Drone.io limits 15mins for a build

Drone.io limits 15mins for a build.Install from wheel package may help you

50


1. curl -L -s https://example.com/wheelhouse.tgz | tar vzxf -

2. export PIP_FIND_LINKS=./wheelhouse3. pip install sphinx sphinx-intl transifex-client==0.8run.sh

ex. https://bitbucket.org/sphinxjp/docutils-translation/

TRAP: Version of transifex-client

transifex-client 0.11b3 is not stable (for me)Especially for Windows users

If you have encountered trouble with using newer version, please try:

51


$ pip install "transifex-client=0.8"

Drone.io only list-up repositories which you have admin permission.

Prepare a empty repository to create a drone.io project.

52

TRICK: Preparing Drone.io project5. Tips, tricks, traps

Examples

Sphinx-1.3 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc13/admin

Sphinx-1.4 doc for "ja" translationhttps://drone.io/bitbucket.org/shimizukawa/sphinx-doc14/admin

Docutils doc for "ja" translationhttps://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin

53


https://drone.io/bitbucket.org/sphinxjp/docutils-translation/admin








Questions?@shimizukawa

Grab me at Night Market.

54

Thanks :)

55

Date post:	31-Jul-2015
Category:	Technology
Upload:	takayuki-shimizukawa
View:	950 times
Download:	0 times

Easy contributable internationalization process with Sphinx (PyCon APAC 2015 in Taiwan)

Technology