Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
OpenNMT: Open-Source Neural Machine Translation with Torch Mathematical Toolkit (github.com/opennmt)
59 points by dragonsh on July 2, 2020 | hide | past | favorite | 9 comments


Hi all, I'm one of the original authors of OpenNMT and an nlp prof, always nice to see it trending :). I work with Hugging Face (https://huggingface.co/) now, we do similar things for NLP generally as well as supporting NMT applications.

Main Versions (supported by folks at systran): https://github.com/OpenNMT/OpenNMT-tf https://github.com/opennmt/opennmt-py

Happy to answer any questions about NLP, neural models, open-source ML.


I noticed that your swift libraries will be ready so that refined attention based models can be put on device for iOS . Will you also support android also ?


Looks cool, but it hasn't been updated in a bit and as they note, the Torch framework is no longer maintained. There is a PyTorch alternative that appears more active: https://github.com/OpenNMT/OpenNMT-py.


As someone who has work with all three NLP toolkit: huggingface, openmt-py and fairseq. I always have trouble juggling through the heavy abstraction of openmt-py.

For example in openmt-py you need to write fields, reader and raw datasets before you even load into their complex dataset class. Each item is heavily abstracted through several layers of classes. I understand this improve code reuse, but introduce a huge steep curve for newcomer.

Huggingface approach on the other hand is slightly more "messy" [2] but easier to understand and add your own tweak.

[1] https://github.com/OpenNMT/OpenNMT-py/blob/master/onmt/bin/p...

[2] https://github.com/huggingface/transformers/tree/master/src/...


There are many high-performing machine translation toolkits that use PyTorch these days:

- OpenNMT-py (OpenNMT, but Python!)

- fairseq

- JoeyNMT

In addition, you can also try

- tensor2tensor (Tensorflow)

- Sockeye (MXNet / Gluon)


Are any of these any good at JP/EN? Google translate is a big thorn in my privacy behind.


You may be lucky - people from NTT have published a model based on JParaCrawl - a large JP/EN parallel corpus, which can be used together with fairseq

http://www.kecl.ntt.co.jp/icl/lirg/jparacrawl/


This is much more used, it's based on FAIRSeq Transformer: https://github.com/modernmt/modernmt. Plus, it's a publicly funded project


How many sentence examples do you need to train one of these?

How much compute?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: