migration to go mod, moving from standard library http handler to echo #17

joicemjoseph · 2020-07-16T16:57:04Z

No description provided.

joicemjoseph · 2020-07-16T18:10:52Z

joicemjoseph · 2020-08-09T03:32:47Z

Good to go

subins2000 · 2020-08-31T17:27:55Z

I don't know Go, is echo a HTTP library like express in nodejs ? Would it be possible to add an authentication to only allow some people to add words with /learn endpoint ?

joicemjoseph · 2020-09-01T06:08:13Z

Yes, It is possible. We can do it without any extra libraries. They have a middleware. We can do the same without any third party library. On request, encode username and password in base64 format and send it on Authentication header, decode and string match credentials and can proceed.

subins2000 · 2020-09-02T21:45:30Z

I tried the branch, but getting this error on learn handler.

curl 'http://localhost:8080/learn' -H 'Origin: http://localhost:8080' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://localhost:8080/' --data-raw $'{"text":"\u0d15\u0d4d","lang":"ml"}'

gives :

{"message":"unable to find language"}

subins2000 · 2020-09-03T07:10:42Z

main.go

 	varnamdConfig = initConfig()
 	startedAt = time.Now()
+
+	runtime.GOMAXPROCS(runtime.NumCPU())
+	flag.Parse()


This should be called before initConfig() is called, otherwise flag values won't be set

I will check this today.

subins2000 · 2020-09-03T07:13:12Z

Running varnamd -enable-download ml,hi -sync-interval 1 doesn't seem to do anything too. Idk what's the expected working @navaneeth

joicemjoseph · 2020-09-03T11:05:52Z

I tried the branch, but getting this error on learn handler.

curl 'http://localhost:8080/learn' -H 'Origin: http://localhost:8080' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Referer: http://localhost:8080/' --data-raw $'{"text":"\u0d15\u0d4d","lang":"ml"}'

gives :

{"message":"unable to find language"}

try
curl -XPOST 'localhost:8080/learn' -d '{"word":"\u0d15\u0d4d","lang":"ml"}' -i -H 'Content-Type: application/json'

PS: Make sure you have mentioned content-type

subins2000 · 2020-09-03T11:22:24Z

Didn't work, same error happens. Can you try it out in varnamd's UI and see if it works on your side ?

joicemjoseph · 2020-09-03T11:47:06Z

Didn't work, same error happens. Can you try it out in varnamd's UI and see if it works on your side ?

There was an issue, which I mistaken for a json element key. I resolved it. But I found out that request's content type is not 'application/json' but 'form-url-encoded'.

The request is made from https://github.com/varnamproject/varnamd/blob/master/ui/javascripts/varnam.js#L77 It is in the minified form. @subins2000 could you send me a patch by adding content type as 'application/json' ?

joicemjoseph · 2020-09-03T11:58:50Z

moved to draft as I found a bug in download content encoding

…ally

subins2000 · 2020-09-03T17:34:15Z

Can confirm, learn endpoint now gives a "success" response. Thanks for the fix!

joicemjoseph · 2020-09-15T12:32:26Z

There is a default value in varnamd, without any config file, it will run on port 8080.

as a workaroud, we can add env variable as well as config flag.

(Thanks for pointing out ibus-varnam's varnamd dependency. I forgot that)

subins2000 · 2020-09-15T12:53:25Z

Perhaps allow usage of flags that'll override the config file ?

…o feat/gomod-echo

…ual word, pattern mapping to channel

joicemjoseph · 2020-09-17T17:52:04Z

Perhaps allow usage of flags that'll override the config file ?

1f62ad1 resolved this

Add endpoint for downloading Varnam Symbol Table, Removing Word, Cache Improvements

* Add endpoint for uploading a file and learning from it * Add /learn/upload enpoint for files to be uploaded and learnt * Make /learn endpoints under auth * refactored code * Refactor, move functions to better suited files, use temp dir for upload * Disable auth by default * Change tmpdir Co-authored-by: Joice <[email protected]>

* Add /packs, /packs/:langCode endpoint, basic packs structure * Funcs should return errors too * Add /packs/:langCode/:packIdentifier endpoint for downloading a pack * Add endpoint to download pack from upstream * Enable internal API by default * Add learning from pack file * Add endpoint to get a pack version info & a /download to download that pack

navaneeth · 2020-10-05T16:25:19Z

There are so many changes. Not sure I have gone through and tested fully. @joicemjoseph , @subins2000 tested this?

joicemjoseph · 2020-10-05T16:30:01Z

Yep. There are a lot of changes. Yet more changes are on pipeline. We have tested. But, Subin have made some PR to my fork. I will merge them. Then we can merge this PR. How that would be.
??

navaneeth · 2020-10-05T16:36:54Z

@joicemjoseph Sure. That makes sense

* Add different endpoints for pack & diff pack versions * Show 404 if no pack found * packs.json should be updated when a new pack is downloaded from upstream * Disallow pack download if already installed * Test upstream URL * Refactor better * Assign small-caps JSON property names

subins2000 · 2020-10-05T17:43:51Z

@navaneeth This PR is now complete. It'd be great if you can give org access to both of us so that we can work straight on varnamd (with PRs of course) skipping the fork of a fork changes.

* Remove usage of packs.json and use pack.json inside different packs * Update pack.json when packs are downloaded * Packs are now trained files to import from and not files to learn from * Show error if pack import failed, don't update pack.json then * Cache packs info for faster requests and avoid recurrent disk reads * Compress pack downloads into gzip, considerable filesize decrease * Use new vpf extension for pack file * division is the correct way to convert to int

.vlf - Varnam Learnings File

subins2000 · 2020-11-28T19:58:57Z

We've been running this changed varnamd here and on the desktop app. 90% of these changes have been tested and works well.

Recommending this to be merged.

joicemjoseph marked this pull request as draft July 16, 2020 16:57

joicemjoseph marked this pull request as ready for review July 17, 2020 05:06

joicemjoseph force-pushed the feat/gomod-echo branch from 6298cdc to 3ab20c5 Compare July 17, 2020 05:51

joicemjoseph added 3 commits July 20, 2020 21:22

added go mod, golang ci

2f84db7

migrated to echo

b55980f

fix: issues in echo server config, html, css files formatted

070c3f9

joicemjoseph force-pushed the feat/gomod-echo branch from 3ab20c5 to 070c3f9 Compare July 20, 2020 15:54

corrected wildcard for handlers

a3889f4

subins2000 reviewed Sep 3, 2020

View reviewed changes

made flag parsing before initializing configs

196c62c

fixed json key

39ae0a0

joicemjoseph marked this pull request as draft September 3, 2020 11:58

joicemjoseph added 2 commits September 3, 2020 21:19

fixed lint errors, added Content-Type as json for learn endpoint manu…

705e8ec

…ally

fixed content encoding issue in gzip content encoding

13622db

joicemjoseph marked this pull request as ready for review September 3, 2020 17:26

corrected error check pattern

547961c

subins2000 mentioned this pull request Sep 10, 2020

Endpoint for adding/training a word #19

Open

joicemjoseph added 2 commits September 11, 2020 17:35

added cache for tl

96b908d

added caching for rtl

a6e274c

joicemjoseph added 6 commits September 16, 2020 22:35

fixed undefined variable error

d7d6086

Merge branch 'feat/gomod-echo' of github.com:joicemjoseph/varnamd int…

232c476

…o feat/gomod-echo

corrected endpoints and rest api structure for train endpoint

c343925

added handler to train in bulk, changed structure of training individ…

8a3a331

…ual word, pattern mapping to channel

added basic auth for training handles

3f7f971

rolled back address to flags - p and host respectively

1f62ad1

subins2000 mentioned this pull request Sep 24, 2020

Authenticated endpoint for removing a word : /remove #21

Open

subins2000 and others added 9 commits September 28, 2020 00:23

Add endpoint for downloading vst

ac3afee

Fix varnamproject#24 : add endpoint to download scheme file (vst)

e65b78b

Add authenticated endpoint for deleting a word varnamproject#21

8b42528

Clear cache after delete or train

7967718

Simplified code

25338ae

Merge pull request #4 from subins2000/vst-download

aee520f

Add endpoint for downloading Varnam Symbol Table, Removing Word, Cache Improvements

fixing lints

da3e5ef

subins2000 and others added 4 commits October 8, 2020 15:55

Fix packs dir creation permission

fceeafd

.vlf - Varnam Learnings File

27570aa

Merge pull request #10 from subins2000/vlf

837e219

.vlf - Varnam Learnings File

subins2000 approved these changes Nov 28, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migration to go mod, moving from standard library http handler to echo #17

migration to go mod, moving from standard library http handler to echo #17

joicemjoseph commented Jul 16, 2020

joicemjoseph commented Jul 16, 2020

joicemjoseph commented Aug 9, 2020

subins2000 commented Aug 31, 2020

joicemjoseph commented Sep 1, 2020

subins2000 commented Sep 2, 2020

subins2000 Sep 3, 2020

joicemjoseph Sep 3, 2020

subins2000 commented Sep 3, 2020 •

edited

Loading

joicemjoseph commented Sep 3, 2020 •

edited

Loading

subins2000 commented Sep 3, 2020

joicemjoseph commented Sep 3, 2020

joicemjoseph commented Sep 3, 2020

subins2000 commented Sep 3, 2020

joicemjoseph commented Sep 15, 2020 •

edited

Loading

subins2000 commented Sep 15, 2020

joicemjoseph commented Sep 17, 2020 •

edited

Loading

navaneeth commented Oct 5, 2020

joicemjoseph commented Oct 5, 2020

navaneeth commented Oct 5, 2020

subins2000 commented Oct 5, 2020

subins2000 commented Nov 28, 2020

migration to go mod, moving from standard library http handler to echo #17

Are you sure you want to change the base?

migration to go mod, moving from standard library http handler to echo #17

Conversation

joicemjoseph commented Jul 16, 2020

joicemjoseph commented Jul 16, 2020

joicemjoseph commented Aug 9, 2020

subins2000 commented Aug 31, 2020

joicemjoseph commented Sep 1, 2020

subins2000 commented Sep 2, 2020

subins2000 Sep 3, 2020

Choose a reason for hiding this comment

joicemjoseph Sep 3, 2020

Choose a reason for hiding this comment

subins2000 commented Sep 3, 2020 • edited Loading

joicemjoseph commented Sep 3, 2020 • edited Loading

subins2000 commented Sep 3, 2020

joicemjoseph commented Sep 3, 2020

joicemjoseph commented Sep 3, 2020

subins2000 commented Sep 3, 2020

joicemjoseph commented Sep 15, 2020 • edited Loading

subins2000 commented Sep 15, 2020

joicemjoseph commented Sep 17, 2020 • edited Loading

navaneeth commented Oct 5, 2020

joicemjoseph commented Oct 5, 2020

navaneeth commented Oct 5, 2020

subins2000 commented Oct 5, 2020

subins2000 commented Nov 28, 2020

subins2000 commented Sep 3, 2020 •

edited

Loading

joicemjoseph commented Sep 3, 2020 •

edited

Loading

joicemjoseph commented Sep 15, 2020 •

edited

Loading

joicemjoseph commented Sep 17, 2020 •

edited

Loading