Compare commits

...

775 Commits

Author SHA1 Message Date
apoltavtsev
4f31406bfe fix(APPINT-33909) tREST as OSGI published by CI failed to run in Runtime with ClientHandlerException 2021-10-19 06:05:13 +02:00
jzhao
06462122bb feat(TDI-45864):Support File components with ORC file format (DI) (#6722)
* feat(TDI-45864):Support File components with ORC file format (DI)

* feat(TDI-45864):Date type read/write issue

* feat(TDI-45864):Double/Float from DB to ORC issue

* feat(TDI-45864):sync orc-core version
2021-10-14 16:23:24 +08:00
jzhao-talend
2c3eafb7c2 chore(TDI-46682):twebservice remove useless urlpath 2021-10-14 12:44:38 +08:00
jzhao
eef575ea44 fix(TDI-46682):Can't send e-mails via smtp.office365.com need to upgrade mail-1.4.7.jar to latest version (#6721)
* fix(TDI-46682):Can't send e-mails via smtp.office365.com need to upgrade
mail-1.4.7.jar to latest version

* fix(TDI-46682):update for tPOP/tFileInputMail/tVtigerCRMXX/tWebserviceXX
2021-10-14 12:24:31 +08:00
chmyga
7be4c6a799 fix(TDI-46777): tBigQueryOutput 24 hour format (#6689) 2021-10-13 17:45:21 +03:00
wang wei
f852743538 fix(TDI-46774): tS3Put doesn't copy the whole file over if using cache (#6714) 2021-10-12 10:38:19 +08:00
vadym-drokov
f590b9214d APPINT-33842: NPE in tDB*** when selecting "data source alias" (#6705) 2021-10-08 11:51:04 +02:00
Dmytro Sylaiev
3544beccec fix(TDI-46759): Handle booleans as digits for OB (#6688) 2021-10-08 12:29:34 +03:00
Dmytro Grygorenko
195c2997d2 fix(TDI-46616): CVE - update "commons-io" to 2.8.0 (#6631) 2021-10-07 10:30:14 +03:00
Dmytro Grygorenko
74f1c89f0b fix(TDI-46084): align versions of "jersey-core" and "jersey-client". (#6647)
* fix(TDI-46084): align versions of "jersey-core" and "jersey-client".

* fix(TDI-46084): add "jsr311-api" library.
2021-10-07 10:09:24 +03:00
apoltavtsev
9134549c94 fix(APPINT-33694) Route Splitter is not working with "XPATH" (#6702) 2021-10-05 15:08:27 +02:00
Emmanuel GALLOIS
9e472b477a feat(TDI-46769): bump to component-runtime 1.37.0 (#6673) 2021-10-04 10:52:15 +02:00
bhe-talendbj
41516246d1 fix(TUP-31095): fix CWE-95 (#6695) (#6697) 2021-09-30 11:41:56 +08:00
sbliu
f9fc607e59 fix(TUP-32947) fix tuj compile error, fix unit test. (#6691)
fix unit test: set parameter with correct type.
2021-09-30 09:38:52 +08:00
sbliu
59ace742af fix(TUP-32947): Handle concatenating context variables, build correct update sql with real table name. (#6674) 2021-09-28 16:39:22 +08:00
Dmytro Grygorenko
1e35baefb1 fix(TDI-46614): CVE - update "commons-compress" to 1.21 (#6584)
* fix(TDI-46614): CVE - update "commons-compress" to 1.21

* fix(TDI-46614): version format corrected.

* fix(TDI-46614): keep version for "checkArchive" lib.
2021-09-27 17:13:34 +03:00
Jane Ding
bbe4460cd4 fix(TUP-33067):tAmazonAuroraInput and tAmazonAuroraOutput list multiple (#6678)
different version driver moudules
https://jira.talendforge.org/browse/TUP-33067
2021-09-27 21:06:48 +08:00
zyuan-talend
35e6bf01f9 fix(TUP-32758):show in connection dropdown and keep the built-in while propertytype + db version are compatible. (#6656) 2021-09-26 15:04:31 +08:00
Jane Ding
4191ba8730 fix(TBD-13063):got wrong module for tAmazonAuroraInput (#6668)
https://jira.talendforge.org/browse/TBD-13063
2021-09-22 15:52:48 +08:00
AlixMetivier
22cad3b97a feat(TBD-12334): add run submit mode to DBR (#6629) 2021-09-17 10:03:19 +02:00
jzhao
87fbabdccd fix(TDI-46527):Dataloss with Parquet components for timestamp field with nano seconds(#6627) 2021-09-17 13:37:14 +08:00
clesaec
b62c16ff6a TDI-46721 - shorter generic code (#6640)
* TDI-46721 - shorter generic code
2021-09-16 12:54:51 +02:00
wang wei
32408cd9a7 fix(TDI-46252): not add that parameter as some case not work, and user can set it self as no better way to control it(#6648) 2021-09-16 10:24:22 +08:00
Wei Hua
94a80b55d4 APPINT-33503 '@' not generated with tRESTResponse after migration. (#6649)
- Add migration task to set default value
2021-09-15 20:47:28 +08:00
Jill Yan
c7a9cc1145 APPINT-33092 omit ds aliase if not set (#6635)
* APPINT-33092 omit ds aliase if not set

* APPINT-33092 fix conflict

* APPINT-33092 revert

* APPINT-33092 revert

* APPINT-33902 fix NPE

Co-authored-by: jillyan <yan955599@gmail.com>
2021-09-15 18:33:51 +08:00
jzhao
85ed098bcb fix(TDI-46511):[7.3.1] tMomInput: "Sleeping time" parameter is not used in code generated for Websphere MQ (#6602) 2021-09-15 11:27:55 +08:00
wang wei
25637d3857 fix(TDI-45468): support streaming append (#6623) 2021-09-15 11:05:16 +08:00
Dmytro Grygorenko
b3774b643b fix(TDI-46612): add ERROR_MESSAGE to the DBRow components. (#6626) 2021-09-14 16:40:51 +03:00
jzhao
51d6fb4cac fix(TDI-46315):add migration task for changing the value of one dbversion item (#6606)
* fix(TDI-46315):add migration task for changing the value of one
dbversion item

* fix(TDI-46315):sap component and sap metadata have different module
name/mvnurl.

Co-authored-by: Hongchun Yi <hcyi@talend.com>
2021-09-14 14:50:39 +08:00
wang wei
37ae765116 fix(TDI-46252): fix tAmazonAuroraRow (#6634) 2021-09-14 09:56:38 +08:00
Dmytro Sylaiev
1b891d23f8 fix(TDI-46610): Add missing logout before disconnect FTPS FTPGet (#6576) 2021-09-13 16:35:27 +03:00
vadym-drokov
26c3b77921 APPINT-33638: Datasource ignored when DB components in Joblet (#6612)
* APPINT-33638: Datasource ignored when DB components in Joblet

* Compilation error is corrected

Co-authored-by: apoltavtsev <apoltavtsev@gmail.com>
2021-09-13 11:25:11 +02:00
wang wei
6af4903291 fix(TDI-46348): studio fix for delta format (#6574) 2021-09-13 11:42:12 +08:00
pyzhou
f3a1279436 feat(TDI-46591):tFTPGet create infinite folder (#6582) 2021-09-12 07:38:45 +08:00
Emmanuel GALLOIS
667e43c56e feat(TDI-46568): bump to component-runtime 1.36.1 (#6625)
* feat(TDI-46568): bump component-runtime to 1.36.0
* feat(TDI-46568): bump to component-runtime 1.36.1
2021-09-10 12:48:13 +02:00
Wei Hua
6120adbd1e APPINT-33503 '@' not generated with tRESTResponse after migration. (#6616)
* APPINT-33503 '@' not generated with tRESTResponse after migration.

* set endpoint value
2021-09-10 18:07:15 +08:00
Denis Sergent
63dae01a82 dsergent_fix_APPINT_33649 (#6609)
To avoid class loading issues at runtime (in Talend Runtime), we exclude mssql jar library from OSGi builds.
2021-09-10 10:29:44 +02:00
Emmanuel GALLOIS
52b46db595 feat(TDI-46568): bump component-runtime to 1.36.0 (#6593) 2021-09-10 09:19:56 +02:00
wang wei
5cac10311a fix(TDI-46555): Unrecognized type id_byte[] while doing dynamic schema mapping (#6528) 2021-09-10 14:53:11 +08:00
hzhao-talendbj
fc995fd934 chore(TUP-32664): CVE: commons-compress-[1.19,1.20] (#6605)
* chore(TUP-32664): CVE: commons-compress-[1.19,1.20]

* chore(TUP-32664): CVE: commons-compress-[1.19,1.20]
2021-09-09 10:56:06 +08:00
Olha V
0c1aa7d269 chore(TDI-46132): Update adal4j in tMicrosoftCrm (#6435) 2021-09-08 14:21:52 +03:00
sbliu
58cb31cd0d fix(TUP-32548) Fix Table widget columns for Advanced panel. (#6598) 2021-09-07 15:34:17 +08:00
hzhao-talendbj
2de786b6db fix(TUP-32744): Job latest revision version doesn't match the correct (#6581)
* fix(TUP-32744): Job latest revision version doesn't match the correct
latest SVN revision

* fix(TUP-32744): Job latest revision version doesn't match the correct

* fix(TUP-32744): Job latest revision version doesn't match the correct
2021-09-06 16:18:49 +08:00
hzhao-talendbj
4aa47a90a9 fix(TUP-32790): StackOverflowError occurs by tELTMap (#6592)
* fix(TUP-32790): StackOverflowError occurs by tELTMap when table name is
directly input in Japanese with escaped double quote

* fix(TUP-32790): StackOverflowError occurs by tELTMap
2021-09-06 11:46:27 +08:00
zyuan-talend
018e3e3e06 fix(TUP-32677):set default job version for tRunJob. (#6590) 2021-09-03 10:35:18 +08:00
Jane Ding
69168c56b8 fix(TUP-32671):AS400 Special character (pound sign £) in table column (#6567)
* fix(TUP-32671):AS400 Special character (pound sign £) in table column
names causing errors
https://jira.talendforge.org/browse/TUP-32671

* fix(TUP-32671):AS400 Special character (pound sign £) in table column
names causing errors
https://jira.talendforge.org/browse/TUP-32671

* fix(TUP-32671):AS400 Special character (pound sign £) in table column
names causing errors
https://jira.talendforge.org/browse/TUP-32671
2021-09-02 09:30:31 +08:00
sbliu
cfb02c57c3 Revert "fix(TUP-32548) Fix Table widget columns for Advanced panel. (#6537)" (#6585)
This reverts commit 732e383f8e.
2021-09-01 18:38:23 +08:00
jiezhang-tlnd
1b20a2d08c fix(TUP-30342)Studio misplaces the component dragged from Metadata (#6562) 2021-08-30 10:03:36 +08:00
Tetyana Meronyk
d8af56e14f feat(TDI-42314): tAmazonAuroraRow component (#6577) 2021-08-27 14:25:38 +03:00
jzhao
23070c60a1 fix(TDI-46587):tParquetOutput - DI Job - Dynamic Issue (#6570)
* fix(TDI-46587):tParquetOutput - DI Job - Dynamic Issue

* negative decimal value write issue
2021-08-27 18:43:20 +08:00
pyzhou
7278437430 Pyzhou/tdi 46333 t ftp list name as400 7.3 (#6554)
* fix(TDI-46333): change ListFile to ListName for AS400

# Conflicts:
#	main/plugins/org.talend.designer.components.localprovider/components/tFTPGet/tFTPGet_begin.javajet

* fix compile error in tFTPDelete

* fix compile error in tFTPGet
2021-08-27 11:10:28 +08:00
zyuan-talend
0d1d63b882 fix(TUP-32567): decode hex value in job script. (#6563) 2021-08-26 17:56:29 +08:00
sbliu
732e383f8e fix(TUP-32548) Fix Table widget columns for Advanced panel. (#6537) 2021-08-26 09:41:56 +08:00
sbliu
c7e01ebe67 fix(TUP-31910) Duplicating component with Existing connection resets the connection setting. (#6545) 2021-08-26 09:40:53 +08:00
Dmytro Grygorenko
4b365b194b chore(TDI-46270): improve logging for tFileList component. (#6561) 2021-08-25 16:28:10 +03:00
bhe-talendbj
7f7c963cdc fix(TUP-32682): set category for tck InputSchemaParameter (#6565) (#6568) 2021-08-25 17:42:34 +08:00
pyzhou
586bcb9d23 fix(TDI-46031):tRedshift set default log writer as null (#6560) 2021-08-24 15:20:27 +08:00
wang wei
bac3605a26 fix(TDI-46185): MySQL Connection issues due to TLS 1.2 after java upgrade to 1.8.0_292 (#6306) 2021-08-24 09:07:51 +08:00
vyu-talend
c37faee0d1 Vyu/tdi 46059 missing jars in tservicexxx (#6532)
* fix(TDI-46059):change the mvn url and upload jars

* fix(TDI-46059):change mvn path to official.
2021-08-19 16:49:56 +08:00
sbliu
bca1ab75b5 Revert "fix(TUP-31910) Duplicating component with Existing connection resets the connection setting. (#6487)" (#6541)
This reverts commit 250580cddd.
2021-08-18 10:13:19 +08:00
msjian
e03c026b74 fix(TDQ-19520): fix NPE (#6486) 2021-08-17 07:57:35 +00:00
Dmytro Sylaiev
342a7350be fix(TDI-46496): Hide password property for sftp not pass auth (#6488) 2021-08-16 17:27:17 +03:00
Jane Ding
4cc1dd3de9 fix(TUP-32278)Fix the migration task FixProjectResourceLink execute in (#6525)
every item
https://jira.talendforge.org/browse/TUP-32278
2021-08-12 11:19:06 +08:00
zyuan-talend
ffb98f3f6f fix(TUP-30465): fix the file explorer open on Linux. (#6524) 2021-08-11 17:31:11 +08:00
qiongli
62c2e341c5 fix(TDQ-19637):Judge 'tdqReportRun' in main/sub-job when generate code (#6514)
Co-authored-by: qiongli <qiongli@192.168.1.102>
2021-08-11 15:48:52 +08:00
hcyi
d22a213e38 fix(TUP-32438):Parameter (Component List) is empty but is required. (#6503)
* fix(TUP-32438):Parameter (Component List) is empty but is required.

* fix(TUP-32438):Parameter (Component List) is empty but is required.
2021-08-11 15:01:32 +08:00
hzhao-talendbj
5a3fd2ef23 chore(TUP-32327): remove dup dependency for maven plugins in job (#6501)
chore(TUP-32327): remove dup dependency for maven plugins in job  template
2021-08-11 14:35:24 +08:00
wang wei
f6448c1316 fix(TDI-46475): Deprecate the list of components in 7.3 (#6497) 2021-08-10 15:38:14 +08:00
Jane Ding
b863480a14 fix(TUP-32383):Migration executed at every logon (#6515)
https://jira.talendforge.org/browse/TUP-32383
2021-08-10 11:37:54 +08:00
Emmanuel GALLOIS
19a9126382 chore(studio731): bump component-runtime to 1.35.1 (#6509)
* feat(TDI-46542): fix module inclusion in dependencies.txt when build is java9+
2021-08-09 09:22:49 +02:00
Carlos Badillo
8e9540b70b Revert "fix(TBD-12184):Password field missing for tImpalaRow (#6461)"
This reverts commit e27ab22bd0.
2021-08-09 08:25:35 +02:00
Carlos Badillo
8c0a3390a6 Revert "Cbadillo/fix/tbd 12184 (#6481)"
This reverts commit 36fd68a527.
2021-08-09 08:25:04 +02:00
jzhao
9eac6bc883 feat(TDI-46423):tSQSOutput - Get Amazon SQS Response - "Message ID" (#6477)
* feat(TDI-46423):tSQSOutput - Get Amazon SQS Response - "Message ID"

* feat(TDI-46423):fix migration

* fix(TDI-46423):keep same field name"MessageId" between input and output
2021-08-06 18:25:08 +08:00
zyuan-talend
37a0af7c4e feat(TUP-30465):remove older versions of job. (#6455) 2021-08-06 17:31:13 +08:00
Max
167f9aa41e fix(TBD-12466): httpclient upgraded to 4.5.13 (#6420)
Co-authored-by: Svitlana Ponomarova <sponomarova@talend.com>
2021-08-05 19:36:48 +03:00
sbliu
250580cddd fix(TUP-31910) Duplicating component with Existing connection resets the connection setting. (#6487) 2021-08-05 11:36:09 +08:00
hzhao-talendbj
1f8d842706 chore(TUP-32326): CVE: maven-compat-3.0[3.2.3-3.6.3] (#6470)
* chore(TUP-32326): CVE: maven-compat-3.0[3.2.3-3.6.3]

* TUP-32326 revert esb change
2021-08-04 16:52:30 +08:00
Jane Ding
cc3061d762 fix(TUP-32310):Migration task for checkbox to "true" for existing (#6489)
components will make the components from deactive to active.
https://jira.talendforge.org/browse/TUP-32310
2021-08-04 14:42:38 +08:00
kjwang
da2d50fbe2 TUP-32307 Studio Integration plugin: Support of "Table" values in TCK (#6456)
* TUP-32307 Studio Integration plugin: Support of "Table" values in TCK
components.
https://jira.talendforge.org/browse/TUP-32307
2021-08-04 14:16:30 +08:00
Jane Ding
9b9a1dfaed fix(TUP-32333):Not able to access a repository resource file in TMC (#6480)
https://jira.talendforge.org/browse/TUP-32333
2021-08-03 10:02:33 +08:00
sbliu
cfb02bd135 fix(TUP-32120) tELTMap:left join works as inner join (#6459)
tELTMap:correct update sql when explict join set for table, make explict join parsed it is, but not default inner join
2021-07-30 18:14:12 +08:00
cbadillo1603
36fd68a527 Cbadillo/fix/tbd 12184 (#6481)
* fix(TBD-12184):Password field missing for tImpalaRow

* fix(TBD-12184):Password field missing for tImpalaRow
2021-07-30 10:49:04 +02:00
cbadillo1603
e27ab22bd0 fix(TBD-12184):Password field missing for tImpalaRow (#6461) 2021-07-30 10:39:56 +02:00
sbliu
0776ee5b9f chore(TUP-31799) upgrade plexus-archiver. (#6469) 2021-07-29 19:17:02 +08:00
jzhao
8b1bc0e1ac fix(TDI-46279):tFileInputParquet can't read INT96 type from hive table (#6443)
* provide method to get time millis from nanotime
2021-07-29 15:28:20 +08:00
jiezhang-tlnd
9715a9e018 fix(TUP-31506)Arrow is not in the right direction when there are more (#6457)
than one input component
2021-07-27 16:13:46 +08:00
clesaec
62e441c8d7 TCOMP-1920 : active if fix (#6451)
* TCOMP-1920 : active if fix

* TCOMP-1920 : add assertions

Co-authored-by: jiezhang-tlnd <jie.zhang@talend.com>
2021-07-27 15:53:46 +08:00
Zhiwei Xue
665fd4c320 fix(TUP-32001):Tacokit slow to start (#6369) (#6430)
* fix(TUP-32001):Tacokit slow to start (#6369)

* fix(TUP-32001): fix TCK server readiness checker

Co-authored-by: bhe-talendbj <bhe@talend.com>
2021-07-26 11:01:04 +08:00
Dmytro Sylaiev
fdabd40f24 fix(TDI-46355): Add logout for ftp and ftps before disconnect (#6383)
* A bit reduce code duplication
2021-07-23 11:37:22 +03:00
jiezhang-tlnd
617dbff52b bugfix(TUP-26206):'Refresh Preview' can not work for Json with context (#4480) (#6439)
bugfix(TUP-26206):'Refresh Preview' can not work for Json with context mode

Co-authored-by: hwang-talend <hwang@talend.com>
2021-07-21 17:57:14 +08:00
Dmytro Sylaiev
1740c3626a Dsylaiev/tdi 46367 sftp http proxy enchancement (#6407)
* fix(TDI-46367): Add support for local proxy config for tHTTPRequest

* fix(TDI-46367): Add nonProxy support for ftp,sftp,ftps
2021-07-21 10:27:16 +03:00
sbliu
5b0e0b449c fix(TUP-32255) fix Studio create webservices metadata fail. (#6449) 2021-07-20 21:03:41 +08:00
hcyi
103b26f50b Hcyi/tup 31123 7.3 (#6389)
* feat(TUP-31123):In studio, do not store login credentials on disk for
TAC and reprompt instead.

* feat(TUP-31123):improve for In studio, do not store login credentials on
disk for TAC and reprompt instead.

* feat(TUP-31123):update the message

* feat(TUP-31123):update the messages
2021-07-15 15:57:45 +08:00
wang wei
d7b050ded7 fix(TDI-46396): align net.minidev:accessors-smart and org.ow2.asm:asm:jar for studio (#6414) 2021-07-15 09:51:01 +08:00
sponomarova
d642a8efda fix(TBD-12218): zookeeper cleanup (#6426) 2021-07-14 15:49:01 +03:00
wang wei
85f43e22db fix(TDI-45795): Upgrade components to use the latest version of cxf / remove axis 1 2021-07-14 17:28:25 +08:00
mbasiuk-talend
e6e3581be6 feat(TDI-45914): bigquery stored procedures (#6363)
* feat(TDI-45914): implement feature for Input component

* feat(TDI-45914): implement feature for SQLRow component

* feat(TDI-45914): implement feature for BulkExec component

* feat(TDI-45914): remove die on error, put error message

* feat(TDI-45914): update bulkexec with gson, reuse version

* feat(TDI-45914): fix SQLRow as standalone component statistic return

* feat(TDI-45914): improve logging code pieces

* feat(TDI-45914): remove extra whitespaces

* feat(TDI-45914): fix gson object missing cid

* feat(TDI-45914): fix gson object missing cid for BulkExec

* feat(TDI-45914): add return values for tBigqueryOutput

* feat(TDI-45914): fix BulkExec and Output statistics, remove virtual cid part
2021-07-14 06:36:32 +03:00
Emmanuel GALLOIS
079173a85e chore(studio731): bump component-runtime to 1.34.1 (#6422) 2021-07-13 09:39:58 +02:00
pyzhou
cf4e374d71 fix(TDI-46340):tMongoDB convert bson to string migration (#6387) 2021-07-13 15:36:29 +08:00
wang wei
13f68ebe73 fix(TDI-46393): runtime log log the password field for tcompv0 as we don't filter HIDDEN_TEXT type field (#6410) 2021-07-13 15:31:18 +08:00
bhe-talendbj
667a4a2649 chore(TUP-31163): upgrade daikon to 0.31.12 (#6330)
* chore(TUP-31163): upgrade daikon to 0.31.12

* chore(TUP-31163): upgrade daikon to 0.31.12
2021-07-13 15:27:59 +08:00
jzhao
46a1f26e66 fix(TDI-46081):Oracle BLOB data not ingesting in table when using Dynamic schema (#6421) 2021-07-13 14:49:10 +08:00
chmyga
e16aa8da65 fix(TDI-46165): tBigQueryInput deletes the token properties file for … (#6329)
* fix(TDI-46165): tBigQueryInput deletes the token properties file for OAuth2.0

* Handle auth problems in a different catch block

* fix(TDI-46165): check for 401 error code
2021-07-13 09:36:09 +03:00
ypiel
781a5addc6 feat(TDI-46398): bump tMSCrmInOut to talend-mscrm:3.8-20210707 (#6417) 2021-07-12 14:37:09 +03:00
chmyga
fe44ae77b4 fix(TDI-45907): single insert Bigdecimal (#6384)
* fix(TDI-45907): single insert Bigdecimal

* Set scale for bigdecimals

* fix(TDI-45907): fix indents

* fix(TDI-45907): consider precision is not set

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@globallogic.com>
2021-07-12 11:59:53 +03:00
sbliu
515028bad5 fix(TUP-31869) Studio, TCK components: Table element issues in "Advanced settings" tab.. (#6404) 2021-07-09 10:48:31 +08:00
wang wei
8323cb0c5b fix(TDI-45136): improve tRunJob to support Dynamic context and EXTRA_CLASSPATH (#6321) 2021-07-09 10:42:48 +08:00
vyu-talend
cf137543ff Vyu/tdi 46167 modify containsspace for column (#6392)
* feat(TDI-46167):improve contaionsSpaces for column.

* feat(TDI-46167):reverse the changes to JDBC and ODBC.

* feat(TDI-46167):keep the minimal changes.
2021-07-09 10:12:29 +08:00
vyu-talend
3ca0f5ac72 fix(TDI-46125):add strict reply parsing to FTP components. (#6360) 2021-07-08 16:16:43 +08:00
wang wei
a5320f0b67 fix(TDI-46220): upgrade json-smart to 2.4.7 (#6300) 2021-07-08 14:39:44 +08:00
wang wei
7bebaa9fbf fix(TDI-46321): tFileOutputExcel outputs 1 less number in NB_LINE global varaible (#6365) 2021-07-08 10:18:52 +08:00
Wei Hua
c51a8712dd APPINT-32340 [tESBConsumer 7.3.1] NullPointerException if context variable is not provided (#6397) 2021-07-08 09:29:40 +08:00
Jill Yan
57ba9552b5 check if JSONObject.NULL before convert (#6399)
* check if JSONObject.NULL before convert

* use equals to compare Object

* correct the logic

Co-authored-by: jillyan <yan955599@gmail.com>
2021-07-07 16:12:36 +08:00
jzhao
31ca62fba6 fix(TDI-43795): Hide Single Insert Query action and mark it deprecated (#4588) (#6403)
Co-authored-by: OleksiiNimych <59691550+OleksiiNimych@users.noreply.github.com>
2021-07-07 15:21:09 +08:00
bhe-talendbj
1990b363f3 fix(TUP-31736): add warning message (#6378)
* fix(TUP-31736): add warning message

* fix(TUP-31736): fix warning message
2021-07-07 11:21:48 +08:00
sbliu
b0175e0c6e fix(TUP-31871) Implicit contextload does not work with microsoftsql server and windows authentication + useNTLMv2=true;domain={MYDOMAIN}. (#6351)
update the driver with user selected but not always the default driver
2021-07-07 09:54:36 +08:00
ypiel
3a8ba3d0d0 feat(TDI-46022): mscrm return representation / custom headers (#6394) 2021-07-06 11:44:12 +02:00
hcyi
301d689b33 fix(TUP-30538):Snowflake test connection should be optional. (#6348)
* fix(TUP-30538):Snowflake test connection should be optional.

* fix(TUP-30538):Snowflake test connection should be optional.

* fix(TUP-30538):Snowflake test connection should be optional.
2021-07-05 16:16:22 +08:00
pyzhou
d6293c745a feat(TDI-46112):Adding profile credential provider (#6379)
* feat(TDI-46112):Adding profile credential provider

* Add feature to Buckets components

* typo

* fix bug of finding profile file

* improve code
2021-07-05 15:33:58 +08:00
wang wei
6a6f966c51 fix(TDI-46251): [7.3.1] MsSQL row named "column" results in code (#6309) 2021-07-05 11:32:59 +08:00
Jane Ding
f5474437f2 feat(TUP-23337):Option to disable Job's screenshot creation. (#6317)
* feat(TUP-23337):Option to disable Job's screenshot creation.
https://jira.talendforge.org/browse/TUP-23337

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-23337):Option to disable Job's screenshot creation.
https://jira.talendforge.org/browse/TUP-23337

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-23337):Option to disable Job's screenshot creation.
https://jira.talendforge.org/browse/TUP-23337

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-23337):Option to disable Job's screenshot creation.
https://jira.talendforge.org/browse/TUP-23337
2021-07-05 08:57:02 +08:00
wang wei
408ea73812 fix(TDI-46329): tS3List NullPointerException in R2021-05 (#6347) 2021-07-02 18:33:28 +08:00
bhe-talendbj
045525ac75 fix(TUP-31858): Implicit Context Load - can't find the jdbc driver class (#6343)
* fix(TUP-31858): Implicit Context Load - can't find the jdbc driver class

* fix(TUP-31858): fix data type

* chore(TUP-31858): refactor

* fix(TUP-31858): add missing JDBC driver properties
2021-06-30 09:47:42 +08:00
jiezhang-tlnd
e96de986bb fix(TUP-31904)java.lang.NullPointerException at (#6359)
org.talend.repository.items.importexport.handlers.cache.RepositoryObjectCache.initialize
2021-06-29 10:48:47 +08:00
Laurent BOURGEOIS
5555f862df feat(TBD-10185):Support Datasets API in tMap - Spark Batch - Inner|Left Join / Match model: Unique match (#6313) 2021-06-25 09:51:18 +02:00
hcyi
8770aa2225 fix(TUP-31218):Medata for salesforce:UI show issue when enable Mutual (#6356)
authenticaiton
2021-06-25 14:09:44 +08:00
hcyi
b97d359382 fix(TUP-31608):Output link from one tELTMSSqlMap to another tELTMSSqlMap (#6332)
is not saved after we remove once.
2021-06-24 18:21:09 +08:00
Olha V
4e6a70a0ef chore(TDI-46067): Bump commons-lang3 (#6227)
* chore(TDI-46067): Bump commons-lang3

* bump commons-lang3 to 3.10 in studio-se to align between all
  components

* chore(TDI-46067): Bump commons-lang3

* clean up unused UrlPath

* chore(TDI-46067): Bump commons-lang3

* updated talend-mscrm lib version after rebase
2021-06-24 10:45:53 +03:00
jzhao-talend
c84b02c186 fix(TDI-46297):FCB 'EnableRegionalDisco' is disabled error when using
tMicrosoftCrmInput
2021-06-21 11:55:58 +08:00
wang wei
f2e2d9dd43 fix(TDI-46218): tRedshiftOutput data loss when insert size matches the input size (#6303) 2021-06-17 14:24:33 +08:00
pyzhou
4f362b8cd4 fix(TDI-44917):error message compile error (#6336) 2021-06-16 11:00:13 +08:00
Oleksandr Zhelezniak
838c12ba2d fix(TDI-46248): fix compile error when dieOnError is checked (#6308) 2021-06-14 11:00:56 +03:00
Dmytro Grygorenko
a20f8c75f0 fix(TDI-46065): Redshift - add "Credential Provider" option to XML layout. (#6207)
* fix(TDI-46065): add "Credential Provider" option to XML layout.

* fix(TDI-46065): wording correction.

* fix(TDI-46065): some more XML layout rearrangements.

* fix(TDI-46065): reverted row number changes.

* fix(TDI-46065): fix for existing S3 connection.

* fix(TDI-46065): added fix for tRedshiftBulkExec component.

* fix(TDI-46065): some corrections, migration task.

* fix(TDI-46065): fix imports.

* fix(TDI-46065): corrections to migration task.

* fix(TDI-46065): process checkbox in migration task.

* fix(TDI-46065): fix for tRedshiftOutputBulkExec component (Web identity token)

* fix(TDI-46065): web_* credential transition from OutputBulkExec to OutputBulk component.

* fix(TDI-46065): set credential provider inaccessible; modify migration task.

* fix(TDI-46065): one more change to migration task.

* fix(TDI-46065): reverting recent changes, restoring migration task.

* fix(TDI-46065): hide "Web identity token" option for Redshift components.
2021-06-11 11:13:33 +03:00
jiezhang-tlnd
e6a05e0738 Jzhang/73/tup 31122 (#6311)
* feat(TUP-31122)new Data Center in AWS Australia
https://jira.talendforge.org/browse/TUP-31122

* update login ui

Co-authored-by: jie.zhang <jie.zhang@LT-DDC8R73.talend.com>
2021-06-10 17:59:12 +08:00
Emmanuel GALLOIS
2e5d89b14a feat(TDI-46215): bump component-runtime to 1.33.1 (#6293) 2021-06-10 09:04:33 +02:00
jzhao
a0bf8ea8b7 feat(TDI-45575):Support Parquet File format for regular DI (#6320)
* add talend-parquet lib
2021-06-10 14:43:04 +08:00
pyzhou
cb2d011370 fix(TDI-44917):Fix Compile Error (#6318)
* fix(TDI-44917):Fix Compile Error

* fix(TDI-44917):Fix Compile Error
2021-06-10 11:36:09 +08:00
vyu-talend
fe9f23eee5 Vyu/tdi 45797 s3put enable object lock feature (#6285)
* feat(TDI-45797):add object lock feature to s3put.

* feat(TDI-45797):optimize code.

* feat(TDI-45797):make some changes.

* feat(TDI-45797):fix code generation error.

* feat(TDI-45797):fix context value issue.

* feat(TDI-45797):fix the common file issue in ts3copy.

* feat(TDI-45797):fix the common file path issue in s3copy.

Co-authored-by: Balázs Gunics <bgunics@talend.com>
2021-06-09 11:17:59 +08:00
pyzhou
e7903640b2 fix(TDI-46062):add checkbox Preserve last modified time tFileCopy (#6261)
* fix(TDI-46062):add checkbox Preserve last modified time tFileCopy

fix(TDI-46062): add Junit test

* MigrationTask

* Hide preserve last modified time when check copy directory
2021-06-09 11:14:14 +08:00
ypiel
b6676e4fbd feat(TDI-45155): ms crm support odata expand 7.3 backport (#6312)
* feat(TDI-45155): ms crm support odata expand 7.3 backport

* feat(TDI-45155): Bump lib in MicrosoftCrmOutput_java.xml

* feat(TDI-45155): formatting issue
2021-06-08 16:16:35 +02:00
Dmytro Grygorenko
5ec26c5514 fix(TDI-46109): update XStream to 1.4.17 (#6283) 2021-06-08 09:20:02 +03:00
pyzhou
aee262c30d fix(TDI-46152):tmap die on error issue (#6291)
* fix(TDI-46152):tmap die on error issue

* fix NPE
2021-06-08 10:29:44 +08:00
bkatiukhov
4b68070278 APPINT-32851 Fix cSplitter with JSonPath is not working as expected (#6232)
Co-authored-by: bohdan.katiukhov <bohdan.katiukhov@KBP1-LHP-A00125.synapse.com>
2021-06-07 13:34:47 +03:00
Jill Yan
c7cc06102f Fix/jill/APPINT-32940 (maintenance/7.3.1) (#6286)
* APPINT-32940

* APPINT-32940

correct logic and if (unselectList.size() > 0), or it will add all dependencies into manifest

* APPINT-32940 format

* APPINT-32940 format

* APPINT-32940 compare maven uri

* APPINT-32940

format

* APPINT-32940 refector

* APPINT-32940  refactor

* APPINT-32940 correct check logic

Co-authored-by: jillyan <yan955599@gmail.com>
2021-06-07 18:01:07 +08:00
wang wei
cfe68fa443 fix(TDI-45979): commons-compress-1.18 to 1.19 (#6274) 2021-06-07 10:22:57 +08:00
pyzhou
a961d53357 Pyzhou/tdi 44917 check components error massage 7.3 (#5448)
* fix(TDI-44917):format

* fix(TDI-44917):add error message

* fix tdie and tCreate table

* fix some compile error

* fix tmap

* fix error

* fix toracleOutput

* fix error

* fix compile error tFileInputMSPositional

* add googleStorageConnection.javajet

* add omission (#5909)

* fix bug

* fix compile error of tNetezzaOutput
2021-06-04 09:27:54 +08:00
mbasiuk-talend
f9004ebd4c chore(TDI-46053): upgrade snakeyaml to 1.26 (#6254) 2021-06-03 11:20:44 +03:00
sbliu
e43b872877 feat(TUP-30834) repace axis1 with axis2. (#6152)
remove axis1 export code for export job. ref TUP-19079.
remove dependency to axis1 for repository metadata, remove dependency to salesforce.
for WSDL2JAVAController, still using axis1 to translate wsdl.
2021-06-02 18:28:38 +08:00
chmyga
387871fb59 fix(TDI-46100): connection support (#6253)
* fix(TDI-46100): connection support

* Add reuse connection support to standalone connectors

* fix(TDI-46100): connection support

* Add comment explaining reflection

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@globallogic.com>
2021-05-28 16:21:21 +03:00
hcyi
159cdc9c9d fix(TUP-31553):Hadoop Metadata Wizard when using custom distro dialog (#6258)
box doesnt pop up to import dependencies.
2021-05-28 14:37:43 +08:00
bhe-talendbj
09bcc66d09 chore(TUP-31617): remove commons-compress-1.18 (#6277)
* chore(TUP-31617): remove commons-compress-1.18

* chore(TUP-31617^C update build.properties
2021-05-28 11:42:42 +08:00
hzhao-talendbj
3bd63f5795 feat(TUP-26184): add the same features to filter fields, same as in the (#6213)
* feat(TUP-26184): add the same features to filter fields, same as in the
tMap

* TUP-26184:  some temp code changes

* TUP-26184 add special checkbox for eltmap table column

* TUP-36184 remove useless code

* TUP-26184 remove useless code

* add line back

* TUP-26184 remove some useless code

* TUP-26184 remove useless code

* TUP-26184 remove useless code

* TUP-26184 fix NPE issue

* TUP-26184 remove useless code

* TUP-26184 fix link can't auto change position issue

* TUP-26184 fix link display issue

* TUP-26184 add Enable/disable column name filter

* TUP-26184 fix some NPE errors when filter is on and select below columns

* TUP-26184 change filter icon position to sync with tmap
2021-05-28 09:59:39 +08:00
Jane Ding
32d256d666 fix(TUP-31316):Error connecting to Azure SQL database with Azure Active (#6270)
directory method
https://jira.talendforge.org/browse/TUP-31316
2021-05-26 17:30:09 +08:00
zyuan-talend
b96ee6514b feat(TUP-30343):have the "Export Dependencies" option checked by default.(#6273) 2021-05-26 11:38:25 +08:00
jiezhang-tlnd
fa08aef33c fix(TUP-31228)Netsuite tck guess schema when use existing connection and (#6241)
* fix(TUP-31228)Netsuite tck guess schema when use existing connection and
Token-based login type

* fix(TUP-31228)Netsuite tck guess schema when use existing connection and
2021-05-26 10:05:04 +08:00
Jane Ding
f2325c166d fix(TUP-30849):Improve build Job performance (#6014)
* fix(TUP-30849):Improve build Job performance
https://jira.talendforge.org/browse/TUP-30849

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-31117):Improve performances related to recursive jobs (#6243)

https://jira.talendforge.org/browse/TUP-31117
2021-05-25 18:10:53 +08:00
jiezhang-tlnd
36f23162bb fix(TUP-TUP-31164)Guess schema button on the informix tDBinput component (#6221)
* fix(TUP-TUP-31164)Guess schema button on the informix tDBinput component
returns zero length datatype

* Guess schema button on the informix tDBinput component returns zero
length datatype
2021-05-25 11:28:41 +08:00
Oleksandr Zhelezniak
7118b02042 feat(TDI-45963): after variables (#5933)
* handle specific metakey
* feature uses metadata endpoint in the framework
* extract metadata in javajet
2021-05-24 12:20:24 +03:00
apoltavtsev
d2ae45d2df bugfix(APPINT-33067) Backport fix for NPE 2021-05-21 10:53:43 +02:00
bkatiukhov
dfa91dd61e APPINT-32843 Fix error while deploying a route with tAzureStorageQueue (#6156)
* APPINT-32843 Fix error while deploying a route with tAzureStorageQueue

* Update osgi-exclude.properties

Co-authored-by: Bohdan Katiukhov <bohdan.katiukhov@synapse.com>
2021-05-21 10:14:34 +03:00
jiezhang-tlnd
f75b7895db fix(TUP-31213)tNetSuiteV2019Input failed to guess schema because preview (#6237)
subjob can't work
https://jira.talendforge.org/browse/TUP-31213
2021-05-21 11:16:53 +08:00
apoltavtsev
e05955934f chore(APPINT-32953) fix the NPE when using DI license 2021-05-20 08:51:01 +02:00
SunChaoqun
38acaab6e1 TESB-31720:[7.4.1]Build a job as OSGi will get a jar with only META-INF (#5756) (#6248) 2021-05-19 20:05:39 +02:00
hzhao-talendbj
19300112e8 chore(TUP-29079): remove some urlpath (#6224) 2021-05-19 15:31:27 +08:00
pyzhou
25ace64c68 Pyzhou/tdi 41535 refactor scp components (#6188)
* fix(TDI-41535):change footer

* fix(TDI-41535):connection and close

* fix(TDI-41535):Filelist

* fix(TDI-41535):tSCPDelete

* fix(TDI-41535):tSCPFileExist

* fix(TDI-41535):tSCPGet & tSCPPut

* fix(TDI-41535):tSCPPut remove duplicate

* fix(TDI-41535):tSCPClose bug

* fix(TDI-41535):tSCPTruncate

* fix(TDI-41535):fix public key compile error

* fix(TDI-41535):tSCPFileList count 0 line

* fix(TDI-41535):tSCPGet overwrite warning

* fix(TDI-41535):tSCPGet error message

* fix(TDI-41535):tSCPGet escape space

* fix(TDI-41535):tSCPGet tSCPPut wildCard

* fix(TDI-41535):tSCPGet nb_line

* fix(TDI-41535):tSCPPut error

* fix(TDI-41535):tSCPPut truncate throw Exception

* fix(TDI-41535):jar for scp components

* fix(TDI-41535):add distributionManagement
2021-05-19 14:14:26 +08:00
apoltavtsev
a188cd0e07 fix(APPINT-32953) NPE is corrected 2021-05-19 04:51:05 +02:00
apoltavtsev
e3473f4aa5 fix(APPINT-32953) Routelets are built before Route 2021-05-18 09:02:45 +02:00
Dmytro Sylaiev
7ac39ecd46 fix(TDI-46060): Fix compile error for tFTPFileList (#6219) 2021-05-14 18:19:00 +03:00
sponomarova
bfe5e903c6 fix(TBD-12358): cfx lib change (#6228) 2021-05-14 13:58:54 +03:00
apoltavtsev
e18a8f48a0 fix(APPINT-32995) Root poms installed in CI mode 2021-05-14 10:54:49 +02:00
vdrokov
f7937c3710 Vdrokov fix appint 32987 maintenance/7.3 (#6202)
* APPINT-32905: Issue with Rest service flow

* APPINT-32987: Fix dublicate variable
2021-05-13 12:09:04 +03:00
hcyi
85e8040773 feat(TUP-26747):improve Outline for joblet and subjob. (#6211)
* feat(TUP-26747):improve Outline for joblet and subjob.

* feat(TUP-26747):improve outline for joblet and subjob.

* feat(TUP-26747):improve outline for joblet and subjob.

* feat(TUP-26747):improve outline for joblet.

* feat(TUP-26747):switch to ComponentSettingsView.
2021-05-13 16:48:24 +08:00
qiongli
70908ad2df chore(TDQ-19297): Upgrade 'cxf' to 3.3.10 (#6199) 2021-05-13 16:21:09 +08:00
Max
a7f1809476 fix(TBD-12358): CVE: Update CXF to 3.3.10 (#6158) 2021-05-13 09:13:52 +03:00
zyuan-talend
2d04f97a64 feat(TUP-25494):Correct dialog title. (#6217) 2021-05-13 10:30:41 +08:00
bkatiukhov
3d5992f017 APPINT-32254 Fix bean-validation ignored when deployed in Runtime (#6174)
* APPINT-32254 Fix bean-validation ignored when deployed in Runtime

* Add specific version for validation constraints

Co-authored-by: Bohdan Katiukhov <bohdan.katiukhov@synapse.com>
Co-authored-by: bohdan.katiukhov <bohdan.katiukhov@KBP1-LHP-A00125.mshome.net>
2021-05-12 19:02:36 +03:00
Max
6b11676a66 fix(TBD-12142): CVE: jersey-core-1.4 and jersey-1.9 (#6168) 2021-05-12 13:38:59 +03:00
zyuan-talend
a102775762 fix(TUP-30430): Clone the connection's UNIQUE_NAME for some links (#6194)
instead of generating a new one in joblet node container.
2021-05-12 17:44:27 +08:00
vyu-talend
2721082b75 feat(TDI-45122):add charset to filefetch's parameters (#6149)
* Added encoding option for the upload parameters. Default behavior is unchanged.
Used OPENED_LIST so context parameters can be used.

* feat(TDI-45122):add charset to filefetch's parameters

* feat(TDI-45122):fix issues.

* feat(TDI-45122):fix issue.

Co-authored-by: Balázs Gunics <bgunics@talend.com>
2021-05-12 16:08:09 +08:00
Liu Xinquan
0c9629ef55 fix(APPINT-32593) java.lang.ClassNotFoundException: org.apache.cxf.message.Message (#6200) 2021-05-12 13:55:30 +08:00
ovladyka
14da1383e1 fix(TDI-45910):Jakarta-ORO imported (#6166)
* fix(TDI-45910):Jakarta-ORO imported

* fix(TDI-45910):Jakarta-ORO updated the mvn GAV and required
2021-05-11 18:41:52 +03:00
mbasiuk-talend
01d97c8f63 chore(TDI-45882): bump cxf 3 3 10 (#6167)
* chore(TDI-45882): bump cxf to 3.3.10

* chore(TDI-45882): remove unused cxf version variable
2021-05-11 16:17:21 +03:00
Oleksandr Zhelezniak
59a1b91e4a chore(TDI-45929): bump talend-mscrm (#6101)
* bump in org.talend.libraries.crm
2021-05-11 15:41:26 +03:00
Oleksandr Zhelezniak
e01d4de5c3 fix(TDI-45968): fix commons-codec module name (#6136) 2021-05-11 15:34:58 +03:00
chmyga
d343213ecb fix(TDI-45859): tFileCopy with different fs (#6083)
* fix(TDI-45859): tFileCopy with different fs

* Add force copy parameter

* Copy and delete file instead of moving

* fix(TDI-45859): tFileCopy with different fs

* Add test

* Fix PR comment

* fix(TDI-45859): tFileCopy with different fs

* Fix UI issue

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@globallogic.com>
2021-05-11 14:32:45 +03:00
Oleksandr Zhelezniak
1bd7157e10 fix(TDI-46005): cve bump httpclient to 4.5.13 (#6164)
* bump httpclient to 4.5.13
* fix groupId for talend-bonita-client in javajet
* update httpcore
2021-05-11 13:11:44 +03:00
Dmytro Sylaiev
b68c9ef23f fix(TDI-45824): Bump jersey for tRest (#6137) 2021-05-11 13:03:33 +03:00
Oleksandr Zhelezniak
bf8ae50d77 fix(TDI-45973): cve bump httpclient to 4.5.13 (#6148) 2021-05-11 12:49:09 +03:00
Dmytro Grygorenko
0757392e0a fix(TDI-46004): tEXist components and tXMLRPCInput dependency import. (#6178)
* fix(TDI-46004): fix for dependency import from Nexus.

* fix(TDI-46004): fixed GAV to match the ones from Maven Central.

* fix(TDI-46004): adding all tEXist components.
2021-05-11 10:33:24 +03:00
wang wei
bdb2545a42 fix(TDI-45491): Contextualized configuration properties for S3 component (#6143) 2021-05-11 14:26:35 +08:00
hcyi
3e46ca4dee feat(TUP-26747):Clickable hyperlink from Outline listed component to the job canvas. (#6121)
* feat(TUP-26747):
Clickable hyperlink from Outline listed component to the job canvas.

* feat(TUP-26747):Clickable hyperlink from Outline listed component to the
job canvas.

* feat(TUP-26747):Clickable hyperlink from Outline listed component to the
job canvas.

* feat(TUP-26747):Clickable hyperlink from Outline listed component to the
job canvas.

* feat(TUP-26747):Clickable hyperlink from Outline listed component to the
job canvas.

* feat(TUP-26747):Clickable hyperlink from Outline listed component to the
job canvas.

* feat(TUP-26747):moved the link with editor button to the first one.

* feat(TUP-26747):link with editor if selected component variable.
2021-05-11 09:51:17 +08:00
Denis Sergent
650b50420b Revert "APPINT-32905: Issue with Rest service flow (#6171)" (#6192)
This reverts commit d5386d1114.
2021-05-10 11:36:29 +02:00
kjwang
76137c4b3c Fix:TUP-31429 Fail to add reference project (#6183)
Fix:TUP-31429 Fail to add reference project
https://jira.talendforge.org/browse/TUP-31429
2021-05-10 17:23:41 +08:00
kjwang
371908919b Fix TUP-31096 Could not find or load main class error on the jobs (#6176)
Fix TUP-31096 Could not find or load main class error on the jobs created on the Feature Branch which has #
https://jira.talendforge.org/browse/TUP-31096
2021-05-10 17:22:56 +08:00
Jane Ding
8a20a15f9f fix(TUP-31237):Invalid username or password when creating a Snowflake (#6132)
Metadata Connection with a Snowflake password that has a slash character
https://jira.talendforge.org/browse/TUP-31237
2021-05-10 16:04:03 +08:00
wang wei
40e5c5f7fd fix(TDI-45663): tS3List: Adds Missing File Details (#6146) 2021-05-10 13:39:02 +08:00
wang wei
bcb2d60a99 fix(TDI-45580): Contextualize multipart threshold parameter for S3 components (#6141) 2021-05-10 13:29:50 +08:00
chmyga
3d9d6734c2 feat(TDI-45836): Standalone connectors support (#6018)
* feat(TDI-45836): Standalone connectors support

* Integrate TCK Standalone connectors to studio

* feat(TDI-45836): Standalone connectors support

* remove NB_LINE after var for Standalone connectors

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@globallogic.com>
2021-05-10 11:16:17 +08:00
wang wei
a54823f72d fix(TDI-45821): CVE: json-smart-2.2.1.jar (#6161) 2021-05-10 11:02:24 +08:00
wang wei
2a4167eb4f fix(TDI-45913): Enforce that only Strings, Maps and HashMaps can be loaded from the crcMap file(#6098) 2021-05-10 10:47:42 +08:00
wang wei
a3a53e8447 fix(TDI-45912): Enforce that System path separator character is indeed a character for tRunjob(#6092) 2021-05-10 10:39:09 +08:00
bhe-talendbj
545bc72afa fix(TUP-31346): add default branch for remote uninitialized git repo (#6182)
* fix(TUP-31346): always show selected item

* fix(TUP-31346): select branch
2021-05-08 16:58:54 +08:00
Max
dbc2f213c2 fix/TBD-12115: CVE: derby-10.11.1.1.jar (#6159) 2021-05-06 12:51:23 +03:00
kjwang
2a5cb99f75 Feat:TUP-30377 Move the "Allow specific characters (UTF8,...)" (#6139)
Feat:TUP-30377 Move the "Allow specific characters (UTF8,...) preference setting to project setting.
https://jira.talendforge.org/browse/TUP-30377
2021-05-06 14:46:29 +08:00
ovladyka
73a00f14bb Fix(TUP-30413:Comma missing for tELTMap with multiple inputs, when aliases are used) (#5844) 2021-05-06 10:04:04 +08:00
vdrokov
d5386d1114 APPINT-32905: Issue with Rest service flow (#6171) 2021-05-05 12:31:19 +03:00
Dmytro Sylaiev
2b90106385 fix(TDI-45642): Fix codegen error for Greenplum (#6102)
* fix(TDI-45642): Fix codegen error for Greenplum

* chore(TDI-45642): Add missing empty line
2021-05-05 11:54:00 +03:00
kjwang
cfc6477b33 Revert "TUP-31096 Could not find or load main class error on the jobs created on the Feature Branch which has # (#6082)" (#6169)
This reverts commit 5f5c92a766.
2021-04-30 18:36:19 +08:00
Dmytro Grygorenko
df55122199 fix(TDI-45879): save initial value of NB_LINE to globalMap. (#6076) 2021-04-28 17:19:24 +03:00
ovladyka
98a1bed1e1 fix(TDI-45900): Encoding doesn't work for byte[] type when tFileOutputDelimited use CSV option (#6140)
Updated two javajet files tFileOutputDelimited_begin and tFileOutputDelimited_main
2021-04-28 10:43:13 +03:00
zyuan-talend
194ac012c4 feat(TUP-25494): provide branch search and memory from launcher. (#6138)
* feat(TUP-25494): provide branch search and memory from launcher.
2021-04-28 14:42:22 +08:00
kjwang
5f5c92a766 TUP-31096 Could not find or load main class error on the jobs created on the Feature Branch which has # (#6082)
* TUP-31096 Could not find or load main class error on the jobs created on
the Feature Branch which has #
https://jira.talendforge.org/browse/TUP-31096
2021-04-27 17:57:17 +08:00
OleksiiNimych
d46b547fc9 fix(TDI-45551): SingleStore fix blob type processing (#6084) 2021-04-26 12:32:43 +03:00
Oleksandr Zhelezniak
b05d599f3f fix(TDI-43931) pass full date to independand child job (#6094)
* Convert long as date from context args
* Update file root to be up to date
* Implement fix for tRunJob
* Mention string parse exception in log warn

(cherry picked from commit 87d3fd7c7d)

Co-authored-by: Dmytro Sylaiev <dmytro.sylaiev@globallogic.com>
2021-04-26 11:00:03 +03:00
ypiel
816d395f2d chore: bump to tck:1.32.0 2021-04-22 12:13:35 +02:00
Zhiwei Xue
bfe02643b3 fix(TUP-31246):Inner routine node disappeared after refresh repository (#6130)
view
2021-04-21 17:52:04 +08:00
kjwang
f99d97538f TUP-21090 Support connection component for tacokit (Fix command line (#6124)
* TUP-21090 Support connection component for tacokit (Fix command line
load image error)
https://jira.talendforge.org/browse/TUP-21090
2021-04-20 11:29:36 +08:00
Jane Ding
8d2ff69e40 fix(TUP-30992):CVE: junit-4.11,4.12,4.13 (#6123)
https://jira.talendforge.org/browse/TUP-30992
2021-04-19 19:29:22 +08:00
Zhiwei Xue
f23c9b02ee fix(TUP-31027):[performance] studio will hang when import a special job (#6114) 2021-04-19 15:35:56 +08:00
hcyi
d794cc9a7b fix(TUP-30793):to fix the password problem of RabbitMQ. (#6116) 2021-04-19 15:24:43 +08:00
Jane Ding
ac5cc1ee1d fix(TUP-30992):CVE: junit-4.11,4.13 (#6110)
https://jira.talendforge.org/browse/TUP-30992

Signed-off-by: jding-tlnd <jding@talend.com>
2021-04-16 19:19:19 +08:00
wang wei
70a75cf790 fix(TDI-45577): Job using snowflake dynamic schema fails with special char (#6095) 2021-04-16 18:11:34 +08:00
Jane Ding
db870ecc30 fix(TUP-30992):CVE: junit-4.11,4.13 (#6106)
https://jira.talendforge.org/browse/TUP-30992
fix(TUP-29033):Fail to run testcase in studio and CI
https://jira.talendforge.org/browse/TUP-29033

Signed-off-by: jding-tlnd <jding@talend.com>
2021-04-16 16:30:14 +08:00
Oleksandr Zhelezniak
78f9b554eb feat(TDI-45323): new auth provider web token (#6060)
* replace checkbox with drop-down credential provider list
* migration task
2021-04-15 12:57:13 +03:00
apoltavtsev
7146bdf26c bugfix(APPINT-32288) Propagate "bundleVersion" option 2021-04-15 10:28:05 +02:00
jiezhang-tlnd
4655c0a059 fix(TUP-30992)CVE: junit-4.11,4.13 (#6090) 2021-04-15 16:13:14 +08:00
Max
49658a28d3 fix(TBD-12112): commons-beanutils-core-1.8.0.jar to 1.9.4 (#6012)
* fix(TBD-12112): commons-beanutils-core-1.8.0.jar to 1.9.4

* fix(TBD-12112): dead code elimination

* fix(TBD-12112): additional code cleanup
2021-04-15 10:53:11 +03:00
SunChaoqun
cda46bb231 APPINT-32688:R2021-03 issue with tDB*** using datasource (#6085) 2021-04-15 10:25:13 +08:00
Zhiwei Xue
9df3a48b78 fix(TUP-30791): remove setup code dependencies action for testcase (#6089) 2021-04-14 16:54:13 +08:00
hcyi
9cce21a3bd fix(TUP-30438):Issue when updating snowflake table using tELTOutput. (#5883)
* fix(TUP-30438):Issue when updating snowflake table using tELTOutput.

* fix(TUP-30438):add junts.

* fix(TUP-30438):Issue when updating snowflake table using tELTOutput.
2021-04-14 16:14:48 +08:00
kjwang
174ea89be9 TUP-31145 TCK:Guess schema use an exist connection will overwrite the parameters of component tNetSuiteV2019Input (#6081)
* TUP-31145 TCK:Guess schema use an exist connection will overwrite the
parameters of component tNetSuiteV2019Input
https://jira.talendforge.org/browse/TUP-31145

* TUP-31145 TCK:Guess schema use an exist connection will overwrite the
parameters of component tNetSuiteV2019Input (Fix guess schema issue
cause by TDI-45246)
https://jira.talendforge.org/browse/TUP-31145
2021-04-14 15:26:20 +08:00
kjwang
b4f2124a60 kjwang/Feat TUP-21090 Support connection component for tacokit (#5977)
* kjwang/Feat TUP-21090 Support connection component for tacokit
https://jira.talendforge.org/browse/TUP-21090
2021-04-13 18:06:53 +08:00
Richard Lecomte
5ac16bb7cc TDI-45014 : SFTP auth with password and public key (#6030)
* TDI-45014 : SFTP auth with password and public key

* TDI-45014 : SFTP auth with password and public key

* TDI-45014 : SFTP auth with password and public key

* TDI-45014 : SFTP auth with password and public key

Added parenthesis

* TDI-45014 : SFTP auth with password and public key

Smarter getPassword method
2021-04-13 11:56:15 +02:00
Emmanuel GALLOIS
bc445f065c feat(TDI-45842): bump component-runtime to 1.31.2 (#6029)
* feat(TDI-45842): bump component-runtime to 1.31.1-SNAPSHOT

* feat(TDI-45842): bump component-runtime to 1.31.2

Co-authored-by: jzhao-talend <jzhao@talend.com>
Co-authored-by: mbasiuk <mbasiuk@talend.com>
2021-04-13 12:40:03 +03:00
zyuan-talend
891e6a9d5e fix(TUP-29284): only show proposals for enabled categories(proposal (#6070)
kinds).
2021-04-13 17:20:38 +08:00
hzhao-talendbj
c6e4e79411 fix(TUP-30625): update maven project after convert jobscript to job (#6069) 2021-04-13 14:49:16 +08:00
sbliu
47ffb3d242 feat(TUP-30358) Enhance Data Collector - Route usage details 2021-04-13 10:57:20 +08:00
jiezhang-tlnd
ae30bc1fb3 fix(TUP-30954)CVE_xstream-1.4.15.jar (#6035) 2021-04-13 10:54:41 +08:00
Emmanuel GALLOIS
6b7fce2f78 feat(TDI-45246): do not put in configuration hidden parameters 2021-04-12 12:02:03 +02:00
hcyi
6049577e03 feat(TUP-30291):Add Suggestable support for Table options (List<Row>) in Studio. (#6013)
* feat(TUP-30291):Add Suggestable support for Table options (List<Row>) in
Studio.

* feat(TUP-30291):Add Suggestable support for Table options (List<Row>) in
Studio.

* feat(TUP-30291):improve for the implementation.

* feat(TUP-30291):fix TUP-31031 Add condition, NOT select field, directly
select operator, then throw errors

* feat(TUP-30291):fix TUP-31031 Add condition, NOT select field, directly
select operator, then throw errors

* feat(TUP-30291):fix TUP-31032 [random] Studio can't save field value.

* feat(TUP-30291):fix TUP-31032 [random] Studio can't save field value.
2021-04-12 15:24:26 +08:00
jiezhang-tlnd
186fcafb29 feat(TUP-30381)Support greenplum driver for Greenplum Database in studio (#5995)
* feat(TUP-30381)Support greenplum driver for Greenplum Database in studio
metadata
https://jira.talendforge.org/browse/TUP-30381

Conflicts:
	main/plugins/org.talend.repository/plugin.xml

* remove hard code

* add dbversion for greenplum

* Add REPOSITORY_VALUE for Greenplum components

* set right dbversionString
2021-04-12 10:02:40 +08:00
wang wei
7f3d3b7a59 fix(TDI-45650): [7.3.1] tDeltaLakeOutput- not handling the updates when we are using Dynamic schema(#5908) 2021-04-12 09:35:14 +08:00
Zhiwei Xue
9a11a94043 fix(TUP-30783):Support inner routine with the same name in different (#6065)
custom jar.
2021-04-09 15:50:36 +08:00
Zhiwei Xue
a47de9821f feat(TUP-29952):Change "Assign Routine to" action to "Copy Routine to". (#6059) 2021-04-09 15:49:34 +08:00
Dmytro Sylaiev
f6114ef000 fix(TDI-45642): Count key columns (#6039)
* fix(TDI-45642): Count key columns

* fix(TDI-45642): Fix another regression related to dynamic schema
2021-04-08 19:29:23 +03:00
Zhiwei Xue
9d93ff1652 fix(TUP-30786):Avoid to click finish button more than once when do (#6053)
create custom routine jar/bean jar.
2021-04-08 18:32:57 +08:00
clesaec
51a97c8b24 TDI-45786 - add charset on dynamic (#6027)
* TDI-45786 - add charset on dynamic
2021-04-08 11:38:21 +02:00
Dmytro Sylaiev
29ec16e725 fix(TDI-42478): tFTPConnection : SSL/TLS Client Authentication does not work : no suitable certificate found - continuing without client authentication (#5971)
Co-authored-by: s.bovsunovskyi <s.bovsunovskyi@globallogic.com>
2021-04-08 10:56:55 +03:00
hcyi
6e5e7d1e0a fix(TUP-30731):tELTPostgresqloutput context not work when checked "use (#6021)
update statement without subqueries"
2021-04-08 14:57:12 +08:00
hcyi
5dda69da6a fix(TUP-30793):TCK Datastore on studio Metadata. (#6049) 2021-04-08 11:20:44 +08:00
hzhao-talendbj
e534bed3e0 chore(TUP-27039): Update Commons Compress to 1.19 backport to 7.3 (#5996) 2021-04-08 10:51:44 +08:00
wang wei
56bc8ee766 fix(TDI-45815): CVE: xstream-1.4.15.jar (#6040) 2021-04-08 09:34:57 +08:00
Zhiwei Xue
71413a41dc fix(TUP-30780):Only check used custom jars when run/build Job (#6041) 2021-04-06 15:52:30 +08:00
Dmytro Sylaiev
6240c4331e fix(TDI-45776): Bump slf4j-jdk14 version to make it downloadable (#6017)
* fix(TDI-45776): Bump slf4j-jdk14 version to make it downloadable

* fix(TDI-45776): Apply also for tBonitaInstantiateProcess
2021-04-06 10:48:53 +03:00
Zhiwei Xue
92fac62ac0 fix(TUP-30977):test run map using custom routines and beans doesn't work (#6023)
after switch branch
2021-04-06 12:05:56 +08:00
Laurent BOURGEOIS
8bdca657d4 fix(TBD-11968):CVE commons-collections 3.2.1 (#5866) 2021-04-02 15:43:30 +02:00
bhe-talendbj
ea33bcd37e feat(TUP-30047): Need support of tRunJob with Dynamic Job option enabled on test cases / CI/CD (#6007)
* fix(TUP-30047): Correct classpath for tRunJob in CI mode

* feat(TUP-30047): support running dynamic jobs

* feat(TUP-30047): change location of classpath.jar if running ci test
2021-04-02 16:14:27 +08:00
AlixMetivier
61b2b21833 feat(TBD-11317): allow tS3Configuration to work with assume role in joblet (#5980) 2021-04-01 14:11:27 +02:00
pyzhou
399ae80700 fix(TDI-45834):tFileCopy change module name (#6019) 2021-03-31 16:49:31 +08:00
jiezhang-tlnd
10fd426856 chore(TUP-29381): add dependency for assembly (#5574) (#5839)
* chore(TUP-29381): add dependency for assembly

* add dependency

* add to template

Co-authored-by: hzhao-talendbj <49395568+hzhao-talendbj@users.noreply.github.com>
2021-03-30 15:57:13 +08:00
bkatiukhov
c113df2c41 TESB-32307 tESBConsumer - wrong header content-type (#5976)
Co-authored-by: bohdan.katiukhov <bohdan.katiukhov@KBP1-LHP-A00125.synapse.com>
2021-03-29 10:18:38 +02:00
pyzhou
4eb679c6e9 fix(TDI-45727):CVE jackson-mapper-asl (#5946) 2021-03-26 18:49:01 +08:00
Dmytro Sylaiev
1cf44a07ec fix(TDI-45642): Throw an warning exception when every column is a key… (#5855)
* fix(TDI-45642): Throw an warning exception when every column is a key for update

* Refactor, deduplicate code

* fix(TDI-45642): Fix codegen error for mssql

* fix(TDI-45642): Throw an error for update on duplicate mysql

* fix(TDI-45642): Warn message instead of exception for insert or update
2021-03-25 11:23:49 +02:00
Zhiwei Xue
0bdf41d228 fix(TUP-30813):Add Junits for dependency management feature (#6003) 2021-03-25 09:39:19 +08:00
Laurent BOURGEOIS
c64fec7601 fix(TBD-12776): Fix testAddLog4jToModuleList unit test (#5998) 2021-03-24 11:05:56 +01:00
Oleksandr Zhelezniak
8ab6492011 feat(TDI-45732): extend not dieoneerror area (#5991)
* extend try-block that includes "Input tables (lookups)"
2021-03-24 10:28:51 +02:00
Zhiwei Xue
780ce47ad7 fix(TUP-30779):Custom jar resource unload issue after git pull&merge. (#5982)
* fix(TUP-30779):Custom jar resource unload issue after git pull&merge.

* fix(TUP-30845): fix refreshing NPE
2021-03-24 16:21:11 +08:00
clesaec
9d04099b86 TDI-45772 : unarchiv correction (#5981) 2021-03-23 09:29:59 +01:00
sbliu
e3775bacfe fix(TUP-25417) JDK11: run a job call a child job use twebservice meet error.
using relative path for building parent job(whose child job contains esb cxf components) on jdk11
2021-03-23 14:55:04 +08:00
Jane Ding
0e1a65b82f fix(TUP-30615):Schema Update Detection popping up everytime upon opening (#5973)
the job
https://jira.talendforge.org/browse/TUP-30615

Signed-off-by: jding-tlnd <jding@talend.com>
2021-03-23 11:30:23 +08:00
Oleksandr Zhelezniak
75c51b6dec feat(TDI-45746): fix date context variable (#5943)
* fix the logic of passing date variable to sub jobs
2021-03-19 10:58:02 +02:00
Dmytro Sylaiev
f809f597b4 fix(TDI-45741): Fix checkbox visibility (#5924) 2021-03-19 09:53:22 +02:00
sbliu
97bad0d5ca chore(TUP-30522) add test case for version contains 'SNAPSHOT' 2021-03-18 14:16:56 +08:00
apoltavtsev
a74a54214e fix(TESB-32507) Correct manifest generation for org.talend.esb.authorization.xacml.rt.pep 2021-03-17 20:21:23 +01:00
Jane Ding
bbc2e81686 fix(TUP-30758):tSingleStoreOutputBulkExec can't work (#5964)
https://jira.talendforge.org/browse/TUP-30758

Signed-off-by: jding-tlnd <jding@talend.com>
2021-03-17 09:43:47 +08:00
Max
6bf37640b9 fix(TBD-12011): tDBclose connection deltalake on error - moved to correct package (#5905) 2021-03-16 11:50:49 +02:00
clesaec
b5d8c8d0f3 Clesaec/tdi 45301 t s3 acl (#5829)
* TDI-45301 - ACL canned options added
2021-03-16 10:41:20 +01:00
Oleksandr Zhelezniak
95afb4904e feat(TCOMP-1877): clean cached libs studio-integration (#5961)
* clean cached libs during clean phase for studio-integration plugin
* help to avoid using the out-to-date version of jars in plugin
2021-03-16 10:45:16 +02:00
Zhiwei Xue
322a55e751 feat(TUP-29014): bugfix 2021-03-15 (#5951) 2021-03-15 15:33:36 +08:00
SunChaoqun
028578141e TESB-32453:DemoServiceConsumer/DemoRESTConsumer fail to deploy to (#5952)
runtime with message"[statistics] disconnected"
2021-03-15 15:04:23 +08:00
Dmytro Grygorenko
d7c09e2d71 feat(TDI-45590): migration task for CosmosDB (#5861) 2021-03-14 08:25:45 +02:00
AlixMetivier
24ae727858 feat(TBD-11882): update tCollectAndCheck for BD (#5919) 2021-03-12 16:49:57 +01:00
hzhao-talendbj
1e39f1e09c TUP-30589 fix junit failed (missing jar beanutils-1.9.2 & axis-1.4) (#5939) 2021-03-12 10:30:56 +08:00
SunChaoqun
3c58d86789 TESB-31465:[7.3.1] Studio ESB build performance issue (#5931)
* TESB-31465:[7.3.1] Studio ESB build performance issue

* TESB-32391
[7.3.1] Incorrect OSGi manifest for Routes

* TESB-32391
[7.3.1] Incorrect OSGi manifest for Routes

* TESB-32391:[7.3.1] Incorrect OSGi manifest for Routes

* TESB-32391:[7.3.1] Incorrect OSGi manifest for Routes

* TESB-32391:[7.3.1] Incorrect OSGi manifest for Routes
2021-03-11 23:01:59 +08:00
SunChaoqun
d51d53c3b5 TESB-30792:Upgrade maven plugins (#5932) 2021-03-11 23:01:39 +08:00
Chao MENG
2ca61108c6 fix(TUP-30651): Login page won't be refreshed after delete project (#5940)
https://jira.talendforge.org/browse/TUP-30651
2021-03-11 16:44:43 +08:00
Zhiwei Xue
8c2ea5dd99 feat(TUP-29014):disable rename of custom jar (#5942) 2021-03-11 16:24:51 +08:00
sbliu
ba7c5e45c2 fix(TUP-30257) TMAP - Java & Traces Debug preview not working 2021-03-11 15:03:00 +08:00
zyuan-talend
ba7830ad5c fix(TUP-30589): Mockito cannot mock this class. (#5937) 2021-03-11 11:29:23 +08:00
sbliu
82bc2123f1 fix(TUP-30109) Log4J preferences does not save
backport TUP-26197, needn't force to log4j2 when import old project which is unactive and log4j1.
resolved problem that after restore default value the preference changes cannot be saved on log4j preference page.
2021-03-11 10:54:27 +08:00
pyzhou
5f70c22c91 fix(TDI-45727): replace default IP for proxy (#5917)
* fix(TDI-45727): replace default IP for proxy

* add migration/RemoveDefaultProxyIPTask.java

* add debug

* remove debug
2021-03-10 16:34:54 +08:00
kjwang
9f48439f53 TUP-30648 Migration: tRun job cannot run if Install 731 R02 monthly (#5929)
TUP-30648 Migration: tRun job cannot run if Install 731 R02 monthly patch plus R03 temp patch
https://jira.talendforge.org/browse/TUP-30648
2021-03-10 15:44:50 +08:00
bhe-talendbj
cf25104e30 bugfix(TUP-30378): Talend 7.3.1 tDBInput or tMSSqlInput component will not open query editor (#5886)
* fix(TUP-30378): add exception log

* fix(TUP-30378): add log

* fix(TUP-30378): run open sqlbuilder in background
2021-03-10 10:55:46 +08:00
Jane Ding
ea33684b50 feat(TUP-30169):adapter the tjdbc bulk components in TDI-45487 for (#5862)
* feat(TUP-30169):adapter the tjdbc bulk components in TDI-45487 for
studio and support singlestore database only with a whitelist
https://jira.talendforge.org/browse/TUP-30169

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-30169):adapter the tjdbc bulk components in TDI-45487 for
studio and support singlestore database only with a whitelist
https://jira.talendforge.org/browse/TUP-30169

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-30169):adapter the tjdbc bulk components in TDI-45487 for studio
and support singlestore database only with a whitelist
https://jira.talendforge.org/browse/TUP-30169

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28699):[Bug] The mapping is wrong after dragging (#5514)

tELTMap/tJDBCSCDELT from the created metadata
https://jira.talendforge.org/browse/TUP-28699

Signed-off-by: jding-tlnd <jding@talend.com>

Conflicts:
	main/plugins/org.talend.designer.core/src/main/java/org/talend/designer/core/ui/editor/cmd/ChangeValuesFromRepository.java


Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28699):[Bug] The mapping is wrong after dragging (#5514)

tELTMap/tJDBCSCDELT from the created metadata
https://jira.talendforge.org/browse/TUP-28699

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-30169):adapter the tjdbc bulk components in TDI-45487 for
studio
and support singlestore database only with a whitelist
https://jira.talendforge.org/browse/TUP-30169

Signed-off-by: jding-tlnd <jding@talend.com>
2021-03-09 17:39:55 +08:00
vyu-talend
61c03b2eda fix(TDI-45702):fix the syntax error in eltoutput (#5895) 2021-03-09 17:12:39 +08:00
Zhiwei Xue
127c703af5 feat(TUP-29014): Add only compile code projects function for TDM 2021-03-09 16:08:54 +08:00
Zhiwei Xue
0176cb23ca feat(TUP-29014): Add only compile code projects function for TDM (#5921) 2021-03-09 15:51:29 +08:00
sbliu
d38412eb01 fix(TUP-30250) Not able to share snowflake connection between job and joblet. 2021-03-09 14:25:50 +08:00
Zhiwei Xue
f041bee6b8 feat(TUP-29014): Only build and package beans/routines that are used in Route/Job (#5794)
* feat(TUP-29018): Rearrange single routines into jars

* feat(TUP-29017):Setup routine dependencies for Job

* feat(TUP-29019): Generation and Build: Build routine jars in different
maven projects.

* feat(TUP-29019): Build routine jars in different maven projects

* feat(TUP-29019): fix codesjar cache and update job maven project problem

* feat(TUP-29019): fix codesjar cache and update job maven project problem

* feat(TUP-29019): fix several issues

* feat(TUP-29943):should not have create routines action when object in (#5715)

recycle bin
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29014): refactor codesjar resource cache

* feat(TUP-29019): fix wrong codesjar groupid of ref project in classpath

* feat(TUP-29943):codeJar import items issue (#5725)

* feat(TUP-29943):codeJar import items issue
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):codeJar import items issue
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29014): fix codesjar dependencies can't set required issue

* TESB-31500:[Dependency Management] Only build and package beans that are
used in Route

* feat(TUP-29014): fix NPE of data preview

* feat(TUP-29943): export codeJar with lib modules, import codeJar with (#5740)

lib modules
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29019): improve update codesjar project logic

* feat(TUP-29014):fix assign routine wrong package issue.

* feat(TUP-29014):fix empty class files of codesjar m2 jar issue.

* feat(TUP-29014):fix several assign to action issues and NPE of data
preview

* feat(TUP-29014):support edit codes dependencies for routelet and set
this action read only for testcases

* feat(TUP-29014):fix several import issue and build code jar problem

* feat(TUP-29014): fix nl and some spell issues

* feat(TUP-29943):add codejar check delete reference (#5773)

https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):export and import to deploy libs issue (#5779)

https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29014):fix i18n issue

* feat(TUP-29014):revert the change for building project

* feat(TUP-29014): update ref code projects after resolved dependencies

* Revert "feat(TUP-29014): update ref code projects after resolved dependencies"

This reverts commit 5a93e784e7.

* feat(TUP-29014): support shot class name for custom jars in component

* feat(TUP-29943):rename issues (#5823)

* feat(TUP-29943):rename issues
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):rename innercode; won't store namefor codejar
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):remove old module for properties change and delete
forever
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):check if exist relationship, warn user re-generate all
pom
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):reinstall codejar after inner code rename
https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29014): improve update codesjar project performance and fix
regression

* feat(TUP-29014):fix rename regressions and improve import performance

* feat(TUP-29943):build out job not include inner code items (#5852)

https://jira.talendforge.org/browse/TUP-29943

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29943):disable export action for innercode, and duplicate (#5854)

import issue
https://jira.talendforge.org/browse/TUP-29943
Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29014):fix bd generate code and data preview issues

* feat(TUP-30584):Do not show custom routine/bean jars in TOS (#5898)

* fix(TUP-30597):Fix junit failures caused by dependency management (#5899)

feature

* feat(TUP-29014):fix several git reset issues

* feat(TUP-29014): bugfix 2021-03-05

* feat(TUP-29014): bugfix 2021-03-08

* feat(TUP-29014): bugfix 2021-03-08 2

* feat(TUP-29014):bugfix 2021-03-09

Co-authored-by: Jane Ding <jding@talend.com>
Co-authored-by: SunChaoqun <csun@talend.com>
2021-03-09 12:13:05 +08:00
hcyi
dfcd6e3f2d fix(TUP-30043):Add a new option for tELTOracleMap to generate column alias on selection. (#5806)
* fix(TUP-30043):Add a new option for tELTOracleMap to generate column
alias on selection.

* fix(TUP-30043):Add a new option for tELTOracleMap to generate column
alias on selection.

* fix(TUP-30043):Add a new option for tELTOracleMap to generate column
alias on selection.

* fix(TUP-30043):update title for the new option for tELTOracleMap.

* fix(TUP-30043):Add a new option for tELTOracleMap to generate column
alias on selection.
2021-03-08 18:28:54 +08:00
sbliu
0a7e0e56e4 fix(TUP-30041) do not show warning message if db type column does not show. (#5723) 2021-03-08 14:04:17 +08:00
hzhao-talendbj
bd2e612a44 TUP-30333 tMap with Lookup model Reload at each row freezes Studio (#5817) (#5874)
* TUP-30333 tMap with Lookup model Reload at each row freezes Studio

* TUP-30333 add comments

* TUP-30333 add condition for refresh background
2021-03-08 09:50:43 +08:00
SunChaoqun
d561e36a7e TESB-31465:[7.3.1] Studio ESB build performance issue (#5864)
* TESB-31465:[7.3.1] Studio ESB build performance issue

* TESB-31465:[7.3.1] Studio ESB build performance issue
2021-03-05 18:40:08 +08:00
jiezhang-tlnd
2fd9e82220 TUP-26534 (#4813) (#5769)
Co-authored-by: hzhao-talendbj <49395568+hzhao-talendbj@users.noreply.github.com>
Co-authored-by: hzhao-talendbj <hzhao@talend.com>
2021-03-05 14:43:05 +08:00
kjwang
11a41a331e Fix: TUP-26185 Merge GIT branches - Conflict resolution - the "Compare (#5876)
Fix: TUP-26185 Merge GIT branches - Conflict resolution - the "Compare Result" view does not display differences with a tELTMap component
https://jira.talendforge.org/browse/TUP-26185
2021-03-04 14:12:06 +08:00
Jane Ding
e9fa81a1c8 fix(TUP-30548):Debugger does not work in 7.3 if the installation path (#5894)
* fix(TUP-30548):Debugger does not work in 7.3 if the installation path
contains space
https://jira.talendforge.org/browse/TUP-30548

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-30548):Debugger does not work in 7.3 if the installation path
contains space
https://jira.talendforge.org/browse/TUP-30548

Signed-off-by: jding-tlnd <jding@talend.com>
2021-03-03 22:54:28 +08:00
Emmanuel GALLOIS
9153d30f6e feat(TDI-45704): bump component-runtime to 1.30.0 (#5891)
* feat(TDI-45704): bump component-runtime to 1.30.0
* feat(TDI-45704): fix tests
* feat(TDI-45704): cleanup imports
2021-03-03 10:06:21 +01:00
wang wei
dd863cfd15 fix(TDI-45561): Getting Permission denied error in tFileInputExcel in (#5784)
Co-authored-by: qyliu <qyliu@talend.com>
2021-03-03 10:52:01 +08:00
pyzhou
55d48cfe91 fix(TDI-45418):Upgrade Jackson libraries (#5884) 2021-03-03 10:48:44 +08:00
vyu-talend
7b325e8707 fix(TDI-45613):fix the issue in md5. (#5851) 2021-03-02 18:25:49 +08:00
jiezhang-tlnd
7465b41a34 TUP-27851 Upgrade xstream to xstream 1.4.12 (#5788) 2021-03-01 18:33:37 +08:00
jiezhang-tlnd
79fb201844 chore(TUP-27224)Update Daikon Crypto Utils to 1.15.0 (#5807)
* chore(TUP-27224)Update Daikon Crypto Utils to 1.15.0

* chore(TUP-27224)add migration
2021-03-01 15:50:40 +08:00
pyzhou
45edbf18a1 fix(TDI-45668) CVE ant tfileUnactive (#5868) 2021-03-01 09:30:38 +08:00
bhe-talendbj
f347a16522 chore(TUP-30230): Remove org.talend.libraries.apache.batik (#5766) 2021-02-26 14:39:26 +08:00
apoltavtsev
b9e4faf2bd fix(TESB-32252) Ignore "SNAPSHOT" during dependencies comparison 2021-02-25 07:12:29 +01:00
Chao MENG
1ab7eaeca6 feat(TUP-29801): Improve logon dialog loading time (#5731)
* feat(TUP-29801): Improve logon dialog loading time
https://jira.talendforge.org/browse/TUP-29801

* feat(TUP-29801): Improve logon dialog loading time
https://jira.talendforge.org/browse/TUP-29801

* feat(TUP-29801): Improve logon dialog loading time
https://jira.talendforge.org/browse/TUP-29801
2021-02-24 10:35:43 +08:00
jiezhang-tlnd
069a7b26c3 fix(TUP-30273)tDBOutput component compilation error (#5797)
https://jira.talendforge.org/browse/TUP-30273
2021-02-23 16:33:16 +08:00
pyzhou
6a3651d3b6 fix(TDI-45436):upgrade xstream 2021-02-20 17:59:43 +08:00
Laurent BOURGEOIS
029c0ccb5c fix(TBD-11971):Spark joblets keeps defaulting to HDFS (#5824) 2021-02-12 11:24:12 +01:00
Emmanuel GALLOIS
1382b4efb4 feat(TDI-45225): bump component-runtime to 1.29.1 (#5801)
* feat(TDI-45225): bump component-runtime to 1.28.2 for bouncycastle CVE
2021-02-12 09:55:20 +01:00
hzhao-talendbj
b0d3a70cf3 fix(TUP-30373): Implicit Context with field separator tab "\t" doesn't work (#5825)
* fix(TUP-30373): Implicit Context with field separator tab "\t" doesn't
work anymore

* TUP-30373  code change + add junit
2021-02-09 17:33:32 +08:00
Dmytro Sylaiev
1f20471fbe fix(TDI-45476): Remove adding second list (#5838) 2021-02-09 11:08:34 +02:00
vdrokov
d28ffc5736 TESB-32126: Null pointer exception when accessing job properties (#5837) 2021-02-08 18:01:19 +02:00
vdrokov
146cf1dc0e TESB-31783: Failed to get nested archive for entry BOOT-INF/lib/aws-j… (#5745)
* TESB-31783: Failed to get nested archive for entry BOOT-INF/lib/aws-java-sdk-1.11.848.jar Caused by: java.lang.IllegalStateException: Zip64 archives are not support

* TESB-31783: Failed to get nested archive for entry BOOT-INF/lib/aws-java-sdk-1.11.848.jar Caused by: java.lang.IllegalStateException: Zip64 archives are not supported
2021-02-08 09:26:41 +01:00
hcyi
6c03ca588a fix(TUP-30067):Dataproc Hive metadata not in sync with Hive component. (#5815) 2021-02-08 15:39:19 +08:00
pyzhou
b528ca9a0d fix(TDI-45529):CVE bouncy castle tcloudXXX (#5805) 2021-02-07 16:05:50 +08:00
vyu-talend
888765461c Vyu/tdi 43232 improve where clause for eltoutput m3 (#5774)
* feat(TDI-43232):improve where clause for eltoutput

* feat(TDI-43232):fix some mistakes.

* feat(TDI-43232):fix some issues found by QA.

* feat(TDI-43232):optimize code

* feat(TDI-43232):fix some errors.
2021-02-07 15:17:20 +08:00
ovladyka
ae2f00f1a8 Revert "fix(TDI-45503):Comma missing for tELTMap with multiple inputs, when aliases are used (#5821)" (#5826)
This reverts commit a92fedb9b4.
2021-02-05 11:44:58 +02:00
sbliu
704b63d59c fix(TUP-30186) fix unit test failure. (#5819) 2021-02-05 10:37:00 +08:00
ovladyka
a92fedb9b4 fix(TDI-45503):Comma missing for tELTMap with multiple inputs, when aliases are used (#5821)
Co-authored-by: Oleksandr Vladyka <oleksandr.vladyka@synapse.com>
2021-02-04 16:31:25 +02:00
Hanna Liashchuk
f1a7d2f235 fix(TBD-11964): migration task for use local timezone (#5802)
* fix(TBD-11964): migration task for use local timezone

* fix(TBD-11964): correct date
2021-02-04 13:15:13 +02:00
Dmytro Sylaiev
1702b27493 fix(TDI-45476): Increate tFileOutputMSXML performance (#5741)
* fix(TDI-45476): Increate tFileOutputMSXML performance

* fix(TDI-45476): Change generic type for list to avoid compile error

* fix(TDI-45476): Store sublist into the resourceMap

* fix(TDI-45476): Fix code compile error
2021-02-04 13:02:17 +02:00
bhe-talendbj
cf55315820 fix(TUP-29764): encrypt all PASSWORD fields of job in new format (#5760)
* fix(TUP-29764): migrate all PASSWORD fields of job

* fix(TUP-29764): Resolve comments
2021-02-03 18:02:38 +08:00
vdrokov
d25b125c08 TESB-31044: Data service's endpoint can't be updated if passed as a context variable (#5772) 2021-02-03 09:58:30 +02:00
clesaec
290d9566ed TDI-40364 - change encoding buffer (#5764)
* TDI-40364 : adapt buffer size
2021-02-03 08:23:19 +01:00
pyzhou
0c85bdc4be Pyzhou/tdi 45463 t elt components support delta lake (#5758)
* feat(TDI-45463):tELT components support Delta Lake

* Add delta lake to xml

* add property name

* change logic to hive

* disable Delta Lake for tSQLTemplateMerge

* Revert "disable Delta Lake for tSQLTemplateMerge"

This reverts commit febd9b7e55.

* remove useless code

* correct partition

* correct partition

* add space

* correct mapping and create table

* revert change for tCreateTable_java.xml

* deactive update mode for Delta Lake tELTOutput

* format

* set not-required for partition

* remove test code
2021-02-02 17:36:45 +08:00
jiezhang-tlnd
a4cb0d13d2 fix(TUP-28519):Update org.talend.libraries.apache.google jars (#5199) (#5800)
remove org.talend.libraries.apache.google plugin
https://jira.talendforge.org/browse/TUP-28519

Co-authored-by: Jane Ding <jding@talend.com>
2021-02-02 16:16:20 +08:00
sbliu
502742bad2 fix(TUP-30186) On studio ,java.lang.RuntimeException: Illegal hexadecimal character m at index 0.
fix encode problem when saving job,fix load job problem caused by wrong hex value decode.
handle the wrong migration of hex value, add new migration task to handle wrong jar mvn GAV when import data during last migration, append unit test.
2021-02-02 14:04:52 +08:00
Laurent BOURGEOIS
83a64d3d2c feat(TBD-10921):Add Spark local 3.0.x (#5658) 2021-01-29 18:17:51 +01:00
kjwang
30bbb27e87 Fix TUP-29885 ERROR: Some patches are not compatible with current product (#5714)
* Fix TUP-29885 ERROR: Some patches are not compatible with current
product
https://jira.talendforge.org/browse/TUP-29885
2021-01-29 17:29:56 +08:00
pyzhou
b554d94736 Pyzhou/tdi 45542 t redshift bulk exec support parquet 7.3 (#5795)
* feat(TDI-45542):tRedshiftBulkExec support parquet

* Add STATUPDATE checkbox

* fix show if
2021-01-29 16:07:21 +08:00
Jane Ding
13bbfcca1d fix(TUP-23738):The couchbase icon is not correct when import (#5713)
https://jira.talendforge.org/browse/TUP-23738

Signed-off-by: jding-tlnd <jding@talend.com>
2021-01-29 10:05:04 +08:00
zyuan-talend
6bebbba8ee fix(TUP-23477):fix TreeToTable connection line issue on Mac with BigSur. (#5786) 2021-01-29 09:54:14 +08:00
hcyi
37dc7ca816 fix(TUP-30108):tELTMSSqlInput & tELTMSSqlMap context not recognized after migrating to 7.3.1. (#5751)
* fix(TUP-30108):tELTMSSqlInput & tELTMSSqlMap context not recognized
after migrating to 7.3.1.

* fix(TUP-30108):add more junits .

* fix(TUP-30108):tELTMSSqlInput & tELTMSSqlMap context not recognized
after migrating to 7.3.1
2021-01-28 15:31:23 +08:00
jzhao
242fa6c729 fix(TDI-45400):Dynamic Schema has default length of 100 even after setting to different amount (#5770)
* fix(TDI-45400):Dynamic Schema has default length of 100 even after
setting to different amount.

* fix(TDI-45400):add migration task
2021-01-28 10:57:34 +08:00
wang wei
d1d4dcd7f6 fix(TDI-45432): tBigQueryOutput fails when the checkbox "Create the table if it doesn't exist" is ticked (#5691) 2021-01-26 11:02:06 +08:00
pyzhou
885d14671a fix(TDI-45447): Upgrade bouncycastle to 1.68 2021-01-25 16:17:10 +08:00
clesaec
8fa189bd31 TDI-29308 - Json (#5726) 2021-01-22 07:50:49 +01:00
vdrokov
fe86a1ef43 TESB-31657: AWS SQS not able to deploy to Runtime (#5742) 2021-01-21 18:17:52 +01:00
vdrokov
8751efe56e TESB-31563: Microservice: ClassNotFoundException: org.eclipse.jetty.client.HttpClientTransport (#5670) 2021-01-21 16:55:52 +02:00
mbasiuk-talend
236fb7fc65 fix(TDI-45455): excel output exceeding characters (#5730)
* fix(TDI-45455): truncate exceeding characters, use property

* fix(TDI-45455): add one missing place

* fix(TDI-45455): improve property wording
2021-01-20 20:43:19 +02:00
Chao MENG
c55409088f fix(TUP-30170): [7.3.1] tHbaseInput error with R2020-11 or higher (#5749)
java.lang.String cannot be cast to java.util.List
https://jira.talendforge.org/browse/TUP-30170

Conflicts:
	main/plugins/org.talend.designer.core/src/main/java/org/talend/designer/core/ui/editor/cmd/ChangeValuesFromRepository.java
2021-01-19 17:18:42 +08:00
jzhao
7ead0130ac fix(TDI-45513):tGreenplumGPLoad global variable "RUNTIME" should be Long type(#5746) 2021-01-19 14:28:04 +08:00
clesaec
562d3700b1 TDI-32744 : sub process (#5724) 2021-01-18 11:16:52 +01:00
Emmanuel GALLOIS
3d5fb83763 feat(TDI-45225): upgrade component-runtime to 1.1.29 (#5600)
* feat(TDI-45225): fix test import errors due by bump xbean to 4.18
* feat(TDI-45225): fix deps svc
* feat(TDI-45225): bump component-runtime to 1.28.1
* feat(TDI-45225): fix microservices classloading issues with SPI
2021-01-14 10:24:26 +01:00
zyuan-talend
41f1df71d7 fix(TUP-29784): fix WebService display issues on Mac with BigSur. (#5720) 2021-01-13 17:22:43 +08:00
jzhao
0f7596dcb8 feat(TDI-45144): Support For GreenPlum 6.x (#5716)
* correct and update greenplum driver version
* deprecate postgresql driver
2021-01-12 10:13:46 +08:00
jzhao
520e379e48 feat(TDI-45007 & TDI-31602): Enhance component tGreenplumGPLoad (#5709)
* feat(TDI-45007 & TDI-31602): Enhance component tGreenplumGPLoad

## tGreenplumGPLoad/tGreenplumGPLoad_begin.javajet /
tGreenplumGPLoad/tGgreenplumGPLoad_java.xml

Added guessColumnList so the yaml will contain the columns of the talend
schema, this is extremely useful if the target table has extra columns
(for example to load only the primary keys to a staging table) This will
generate the "  COLUMNS:" section of the YAML file.

Added parameters to the advanced settings, more information for these
can be found at:
https://gpdb.docs.pivotal.io/6-12/utility_guide/ref/gpload.html#topic1__section7
Parameters include:
  LOG_ERRORS:
  MAX_LINE_LENGTH:
  EXTERNAL:
    -SCHEMA:
  PRELOAD:
    -TRUNCATE:
    -REUSE_TABLES:
    -STAGING_TABLE:
    -FAST_MATCH:
  SQL:
    -BEFORE:
    -AFTER:

Changed the way command is built, so the Log4j now displays a command
that can be copy-pasted to the console for testing.
The yaml file was missing from it.
The PythonPath was also not printed to the log4j logs.

Sometimes a database connection was opened and closed without executing
any actions.

Password was never passed to gpload utility so it had to be managed
separately, it is now passed via context variable, which means when
executed by talend it won't prompt for a password and when executing
from console it will prompt for a password.

GPLoad will terminate if there was an error (for example target table
does not exists)

Extra after variables, for the gpload details:
		NB_LINE_INSERTED	Integer
		NB_LINE_UPDATED		Integer
		NB_DATA_ERRORS		Integer
		STATUS				String
		RUNTIME				String

Option to GZip compress the datafile, this will increase CPU usage, but
decreases DISK usage. (50-90% smaller files)
Remove data file after successful execution. If there were *any* errors
the data file is not removed.

## tGreenplumGPLoad/tGreenplumGPLoad_main.javajet
Added byte[] type support. Byte[] is converted to escaped OCTET
characters, which is compatible with all PostgreSQL versions from 8 to
13.

Fixed when newline / escape characters appeared in the data.

Postgresql uses null terminated bytes so those have to be removed from
the data to avoid errors.

## tGreenplumGPLoad/tGreenplumGPLoad_end.javajet
BugFix don't overwrite the actual inserted record number with 0

* feat(TDI-45007 & TDI-31602):Refactor, removed unused code.

* feat(TDI-45007 & TDI-31602):Enhanced log4j logs, so errors / warnings
are easier to spot.
Removed the duplicate log entries for the errors.

* gpload logs can be distinguished propery: gpload says:

* feat(TDI-45007 & TDI-31602):Reworked the PRELOAD and other sections

* feat(TDI-45007 & TDI-31602):Exit loop when external schema is found

* feat(TDI-45007 & TDI-31602):use schema column based the schema dbtype
setting and remove system.out code

* feat(TDI-45007 & TDI-31602): Only rename file when we have an input row
and the filename doesn't end with .gz

* feat(TDI-45007 & TDI-31602): Added proper escape for delimiters such as
vertical tab "\013"

Co-authored-by: Balázs Gunics <bgunics@talend.com>
2021-01-11 15:32:46 +08:00
Dmytro Grygorenko
28ae35dec6 fix(TDI-45440): fix CVE for Hibernate-Core-3.6.10 (#5707) 2021-01-08 10:35:32 +02:00
AlixMetivier
21245a297a fix(TBD-10689): deprecate old BD distrib (#5694)
* fix(TBD-10689): deprecate old BD distrib

* fix UI for tHiveConnection
2021-01-05 09:33:40 +01:00
bhe-talendbj
5131d00675 feat(TUP-29700): initial implement (#5660) 2021-01-05 10:00:22 +08:00
clesaec
f256fc6f38 TDI-45383 : replace without regexp (#5641) 2021-01-04 07:54:31 +01:00
zyuan-talend
30fa41dd46 fix(TUP-29782): fixed ELTDBMap display issues on Mac with BigSur. (#5688)
* fix(TUP-29775): use CTabFolder instead of TabFolder to reslove the display issue in BigSur.

* fix(TUP-29782): fixed ELTDBMap display issues on Mac with BigSur.
2020-12-31 15:32:41 +08:00
zyuan-talend
8a3a1a0289 fix(TUP-29775): use CTabFolder instead of TabFolder to reslove the display issue in BigSur. (#5684) 2020-12-31 15:28:00 +08:00
zyuan-talend
b3468eba73 fix(TUP-29350): fixed the tMap display issues with Big Sur (#5616)
* fix(TUP-29350): fixed the tMap display issues with Big Sur

* fix(TUP-29350): fixed the tMap display issues with Big Sur

* fix(TUP-29641): fixed tAdvancedFileOutputXML display issues with MacOS BigSur (#5647)

* fix(TUP-29774): fixed scroll display issue with many columns.

* fix(TUP-29767): fixed Expression Builder can't show value and can't edit

* fix(TUP-29766): fixed blank expression issue with Tab key.
2020-12-30 16:01:46 +08:00
sbliu
9d9d6b135b fix(TUP-29550) fix problem that "rename" Job in property sheet page duplicates item in Pom file (#5663) 2020-12-29 11:00:15 +08:00
Dmytro Grygorenko
7cbcd2967b fix(TDI-45385): fix CVE for tBonita* components. (#5649)
* fix(TDI-45385): fix CVE for tBonita* components.

* fix(TDI-45385): added missing double quotes
2020-12-28 08:53:33 +02:00
wang wei
918ca363fa fix(TDI-45404): tFileOutputExcel component gives The maximum number of cell styles was exceeded error (#5662) 2020-12-28 10:24:06 +08:00
sbliu
a473b624b7 fix(TUP-29590) fix httpcore sometimes not included when parent job has tRunjob, child job has tHiveConnection , and routine has dependency to httpcore. (#5648) 2020-12-28 10:13:49 +08:00
zshen-talend
1f6128dfd7 feat(TDQ-18757): Support SAP Hana on DQ profiling in Studio (#5596) 2020-12-25 17:22:04 +08:00
SunChaoqun
6fa4c42df1 TESB-31188:[7.3] LinkageError when resolving method (#5611)
"javax.xml.soap.SOAPPart.setContent(Ljavax/xml/transform/Source;)V"
2020-12-23 15:56:15 +01:00
Dmytro Grygorenko
1e61474fc0 fix(TDI-45366): fix for CVE issue with Apache Axis library. (#5642) 2020-12-23 11:33:25 +02:00
pyzhou
d6242e11cd fix(TDI-45377):tGPGDecrypt fail with gpg2 (#5634)
* fix(TDI-45377):tGPGDecrypt fail with gpg2

* fix(TDI-45377):add option

* add migration task
2020-12-23 15:10:10 +08:00
Jane Ding
d007d3740a fix(TUP-28657):Wrong behavior while Job Setting project config to choose (#5632)
From Database
https://jira.talendforge.org/browse/TUP-28657

Signed-off-by: jding-tlnd <jding@talend.com>
2020-12-21 09:20:24 +08:00
mbasiuk-talend
765accb8a3 fix(TDI-45124): enable transfer mode for ftps type (#5520) 2020-12-16 21:46:35 +02:00
mbasiuk-talend
b51c1282f7 fix(TDI-45260): update components with new library version (#5602) 2020-12-16 16:55:54 +02:00
hcyi
c98e18a58c fix(TUP-29072):improve for When renaming link (table) between teltinput and teltmap , the generated SQL query is not updated. (#5628)
* fix(TUP-29072):improve for When renaming link (table) between teltinput
and teltmap , the generated SQL query is not updated.

* fix(TUP-29072):improve for When renaming link (table) between teltinput
and teltmap , the generated SQL query is not updated.

* fix(TUP-29072):improve for When renaming link (table) between teltinput
and teltmap , the generated SQL query is not updated.
2020-12-16 15:49:59 +08:00
bhe-talendbj
e576293e58 fix(TUP-29693): Do not replace maven uri for context inside jarname (#5625)
* fix(TUP-29693): Do not replace maven uri for context inside jarname

* fix(TUP-29693): fix context inside jar name

* fix(TUP-29693): fix context inside jar name

* fix(TUP-2969): fix migrate maven uri for javajet components

* fix(TUP-2969): fix migrate maven uri for javajet components

* fix(TUP-2969): add check for project settings as well

* fix(TUP-29693): Kepp main logic unchanged
2020-12-15 09:50:17 +08:00
pyzhou
55812da6b6 fix(TDI-45346):tFileFetch does not accept 202 (#5612) 2020-12-11 16:25:56 +08:00
Jane Ding
074c13dd4e fix(TUP-28657):Wrong behavior while Job Setting project config to choose (#5597)
* fix(TUP-28657):Wrong behavior while Job Setting project config to choose
From Database
https://jira.talendforge.org/browse/TUP-28657

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28657):Wrong behavior while Job Setting project config to choose
From Database
https://jira.talendforge.org/browse/TUP-28657

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28657):Wrong behavior while Job Setting project config to choose
From Database
https://jira.talendforge.org/browse/TUP-28657

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28657):Wrong behavior while Job Setting project config to choose
From Database
https://jira.talendforge.org/browse/TUP-28657
Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28657):Wrong behavior while Job Setting project config to choose
From Database
https://jira.talendforge.org/browse/TUP-28657

Signed-off-by: jding-tlnd <jding@talend.com>
2020-12-11 11:37:28 +08:00
AlixMetivier
6088e2c2ca fix(TBD-11616): set dataset parameter to true for job already importe… (#5620)
* fix(TBD-11616): set dataset parameter to true for job already imported to 7.3 studios

* refacto

* fix
2020-12-10 17:31:47 +01:00
zshen-talend
1637e6220c Zshen/bugfix/tdq 18791 fix conflict2 (#5619)
* fix(TDQ-18817): support context in Confidence Weight on 7.3 and 7.4

* fix(TDQ-18791): change migration version
2020-12-10 18:09:50 +08:00
AlixMetivier
e96d2af073 fix(TBD-11616): parameterize dataset API utilization (#5562)
* fix(TBD-11616): parameterize dataset API utilization

* apply migration to only 7.3 and 7.4 studios

* fix for future studio versions

* fix bad merge
2020-12-10 09:57:35 +01:00
hcyi
a1e453e79a feat(TUP-25346):tELTTeradataOutput - aliases are mandatory for calculated columns (#5572)
* feat(TUP-25346):tELTTeradataOutput - aliases are mandatory for
calculated columns

* feat(TUP-25346):add junits for calculated columns.

* feat(TUP-25346):NEW RULE for if output column name is different than
output db column.

* fix(TUP-29598):[BUG] The "property setting" setting is missing after
migrating data(TUP-25346)

* feat(TUP-25346):NEW RULE if checked the alias option。

* fix(TUP-29636):[BUG] Studio does not support SQL scripts with alias when
execute update database(TUP-25346)

* feat(TUP-25346):change the junits since NEW RULE if checked the alias
option.

* feat(TUP-25346):remove some junits since no need.

* feat(TUP-25346):format the code

* feat(TUP-25346):format the code
2020-12-10 15:17:08 +08:00
wchen-talend
c8c4a586d0 fix(TESB-31294):move velocity.log to configuration folder (#5586) 2020-12-10 15:00:15 +08:00
Dmytro Sylaiev
89435124fe fix(TDI-45135): Reuse same outputStream when 2 outputDelimited writing to the same file (#5584) 2020-12-09 09:23:50 +02:00
wang wei
ae8e41f36b fix(TDI-45310): trim the label string when log it to avoid the compiler issue(#5589) 2020-12-09 14:09:22 +08:00
Mike Yan
e6fd8b0614 fix(TESB-30734): Backport from patch 7.2.1 (#5468) 2020-12-08 18:07:03 +08:00
Andrii Medvedenko
f4223f7a2b fix(TBD-11724): manually CPing changes from 7.2, TBD-9864 (#5599)
* fix(TBD-11724): manually CPing changes from 7.2, TBD-9864

* fix(TBD-11724): manually CPing changes from 7.2, TBD-9864
2020-12-08 10:36:26 +01:00
wang wei
64dcafb080 fix(TDI-45331): Exit code for SSH component is not working as expected (#5598) 2020-12-08 16:14:40 +08:00
Dmytro Grygorenko
402fe1ffbc fix(TDI-45006): Hide "LIKE" operator from CRM-2016 On-Premise setup. (#5566)
* fix(TDI-45006): Removed unavailable 'CRM_2018' option so that the whole condition could be evaluated.

* fix(TDI-45006): correct way to fix.
2020-12-08 10:00:00 +02:00
hcyi
02c66a7e93 fix(TUP-29366):[bug] studio can rename alias to an existing name. (#5587)
* fix(TUP-29366):[bug] studio can rename alias to an existing name.

* fix(TUP-29366):[bug] studio can rename alias to an existing name.
2020-12-04 16:57:47 +08:00
sbliu
207d1c9635 fix(TUP-29391) fix problem that "Save As" loses changes in the studio for such as job, joblet. (#5560) 2020-12-02 15:38:06 +08:00
Jane Ding
a4d0adb671 fix(TUP-29383):DBInput component miss delta lake when use TDI.license (#5525)
* fix(TUP-29383):DBInput component miss delta lake when use TDI.license
https://jira.talendforge.org/browse/TUP-29383

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-29383):DBInput component miss delta lake when use TDI.license
https://jira.talendforge.org/browse/TUP-29383
Delta lake doesn't support SP, should not list tDBSP

Signed-off-by: jding-tlnd <jding@talend.com>
2020-12-02 14:42:54 +08:00
sbliu
0ce6a06f8e fix(TUP-29224) auto increment the index according to the drag/drop column numbers. (#5555) 2020-12-02 12:00:02 +08:00
hcyi
15cbbf362c fix(TUP-29090):improve since broken some junits . (#5578) 2020-11-30 20:15:08 +08:00
Oleksandr Zhelezniak
c139b0893d fix(TDI-45089): fix date for excel event mode (#5466)
* use date pattern from studio scheme during parse the date column for event mode
* update version of simpleexcel to [2.5-20201119] (the source code of the lib in master branch)
* apply inter-exchange pattern to avoid date rounding
2020-11-30 13:37:53 +02:00
Richard Lecomte
4c2e419bc0 Rlecomte/tdi 45147 t gs copy target issue (#5502)
* TDI-45147 : tGSCopy issue with path

* TDI-45147 : tGSCopy issue with path

Add checkbox to keep legacy behavior by default

* TDI-45147 : tGSCopy issue with path

Add checkbox to keep legacy behavior by default

* TDI-45147 : tGsCopy issue

Resolved conflicts
2020-11-30 11:18:36 +01:00
Dmytro Grygorenko
adc91e4169 fix(TDI-45162): Rearrange imports in tELTPostgresql* and tELTTeradata components. (#5501)
* fix(TDI-45162): Rearrange imports in tELTPostgresql* and tELTTeradata* components.

* fix(TDI-45162): some corrections after branch conflict resolve.

* fix(TDI-45162): more cleanup.
2020-11-30 11:18:25 +02:00
bhe-talendbj
09607ed581 fix(TUP-29166): Remove decorationfigure (#5549)
* fix(TUP-29166): Remove decorationfigure

* fix(TUP-29166): add back arrows

* fix(TUP-29166): remove drawString

* fix(TUP-29166): remove unused method
2020-11-30 10:58:47 +08:00
pyzhou
0a5b925dc4 Pyzhou/tdi 45160 add encording t file unarchive (#5540)
* fix(TDI-45160):add encoding for tFileUnarchive

* fix error

* typo

* add encoding for pass

* upgrade version

* remove import

* remove useless code

* Keep the old behavior

* correct mvn path
2020-11-30 10:38:20 +08:00
ypiel
414ab39434 feat(TDI-44873) : cve - bump olingo odata to 4.7.1 - maintenance
* feat(TDI-44873) : update odata/olingo to v4.7.1

* feat(TDI-44873) : bump to olingo 4.7.1

* feat(TDI-44873) : fix odata lib names

* feat(TDI-44873) : add version to odata MODULE

* feat(TDI-44873) : add version to odata IMPORT name
2020-11-27 17:27:11 +01:00
Dmytro Grygorenko
a269f74a30 fix(TDI-45224): Review and update tJasper* dependencies. (#5545)
* fix(TDI-45224): Review and update tJasper* dependencies.

* fix(TDI-45224): additional dependencies reviewed and updated
2020-11-27 12:15:18 +02:00
clesaec
530814c490 TDI-45193 : dynamic col on tFileInputPositional (#5511)
* TDI-45193 : dynamic col on tFileInputPositional
2020-11-27 10:35:19 +01:00
hcyi
1da69fb285 fix(TUP-29072):When renaming link (table) between teltinput and teltmap , the generated SQL query is not updated (#5467)
* fix(TUP-29072):When renaming link (table) between teltinput and teltmap
, the generated SQL query is not updated

* fix(TUP-29072):When renaming link (table) between teltinput and teltmap
, the generated SQL query is not updated 。

* fix(TUP-29072):add junits

* fix(TUP-29072):add more junits
2020-11-27 17:13:04 +08:00
mbasiuk-talend
3c00488dc8 feat(TDI-44915) access token feature (#5473)
* feat(TDI-44915): integrate Balazs POC

* feat(TDI-44915): implement other BigQuery component with access token

* feat(TDI-44915): update BigQuery bulkexec with new common jet

* feat(TDI-44915): update tGSBucketCreate

* feat(TDI-44915): update tGSBucketDelete

* feat(TDI-44915): update tGSBucketExist

* feat(TDI-44915): update tGSBucketList

* feat(TDI-44915): update connection and close part

* feat(TDI-44915): update tGSCopy

* feat(TDI-44915): update tGSDelete

* feat(TDI-44915): update tGSGet

* feat(TDI-44915): update tGSList

* feat(TDI-44915): update tGSPut

* feat(TDI-44915): use proxy to communicate with GS

* feat(TDI-44915): update code due to PR comments

* feat(TDI-44915): update Input code generation

* feat(TDI-44915): fix tGSPut xml definition

* feat(TDI-44915): update BigQuery Output and generic connection

* feat(TDI-44915): fix _end javajet parts

* feat(TDI-44915): fix bigqueryoutput config mappings

* feat(TDI-44915): fix tGSBuckerCreate dependencies

* feat(TDI-44915): fix PR comments
2020-11-27 10:04:14 +02:00
kjwang
17f54191cf Kjwang/feat tup 28891 temp folder (#5465)
TUP-28891:Shared Studio: Check which functions will write data into
folder "temp" of Studio installation folder from code
https://jira.talendforge.org/browse/TUP-28891
2020-11-27 14:47:48 +08:00
kjwang
ed99155812 Fix : TUP-29358 Performance: It takes more than 1 hour to show "Update Detection" after clicking "detect and update all jobs" icon in a special project (#5548)
* Fix : TUP-29358 Performance: It takes more than 1 hour to show "Update
Detection" after clicking "detect and update all jobs" icon in a special
project
https://jira.talendforge.org/browse/TUP-29358
2020-11-27 14:38:28 +08:00
wang wei
2276f4b51a fix(TDI-45227): talendStats_STATSProcess Error about ELTComponents and Stat&Log tables creation (#5537) 2020-11-27 12:02:29 +08:00
hcyi
da5744d1e5 fix(TUP-29090):[7.2.1] Extra plus '+' signs in generated SQL (#5453) 2020-11-25 10:30:46 +08:00
vyu-talend
3c04002b5e fix(TDI-45159):fix bug in xml of azuresynabe. (#5508) 2020-11-24 15:49:24 +08:00
sbliu
8b67961ade fix(TUP-26486): Can't quickly refresh the GUI when switch format.
when to create azure, quickly refresh the wizard GUI only once when switch format.  fix problem that 'Netsuite/tck: in Metadata credentials fields are not changed based on Auth type selected'.
2020-11-24 15:44:01 +08:00
clesaec
ec914f50fe TDI-45161 : tFile input delimited correction (#5510) 2020-11-24 08:19:06 +01:00
bhe-talendbj
9ab7f01201 fix(TUP-29424): fix junit failures related to Parameter change (#5551) 2020-11-23 11:57:26 +08:00
bhe-talendbj
af79e71c25 fix(TUP-29424): OutputSchemaParameterTest (#5546) 2020-11-23 10:09:59 +08:00
Emmanuel GALLOIS
c50e437c59 feat(TCOMP-1761): Support of complete schema definition in Studio (#5270)
* feat(TCOMP-1761): add nestedProperties for metadata
* feat(TCOMP-1761): update configuration.javajet
* feat(TCOMP-1761): change temporary variable name
2020-11-20 14:17:32 +01:00
Zhiwei Xue
ca07dd16cf fix(TUP-29360): Missing log4j2 jar on user routines (#5529)
* fix(TUP-29360): Missing log4j2 jar on user routines

* fix(TUP-29360): fix switch log4j level problem
2020-11-20 17:12:54 +08:00
ovladyka
64794a596c Fix/TDI45204_IncorrectOutlineFortFileFetch (#5532)
Co-authored-by: Oleksandr Vladyka <oleksandr.vladyka@synapse.com>
2020-11-19 14:08:00 +02:00
pyzhou
ff595fd205 fix(TDI-45167):Close pre Workbook for tFileOutputExcel (#5505) 2020-11-18 11:26:19 +08:00
apoltavtsev
c035091f88 fix(TESB-29553) Publishing a route with cTalendJob from Studio and commandline gives different results 2020-11-12 11:06:56 +01:00
SunChaoqun
3d8c28928a TESB-28330:cConfig/Beans - Support customer groupid and artifact name (#5458)
* TESB-28330:cConfig/Beans - Support customer groupid and artifact name
(additional to custom version)

* TESB-30961:[7.3.1 cConfig] The external jar name and mvn uri is changed
when import a route with cConfig of 721
2020-11-12 16:29:39 +08:00
sbliu
54af03d3ef Revert "feat(TUP-26486) quickly refresh the GUI only one time when switch format."
This reverts commit 1adad9fe93.
2020-11-11 16:05:43 +08:00
sbliu
1adad9fe93 feat(TUP-26486) quickly refresh the GUI only one time when switch format. 2020-11-11 15:38:45 +08:00
Jane Ding
a20ea15f4d fix(TUP-28934):JDBC connection created under folder can not generate the (#5495)
all the components for jdbc when drag to job.
https://jira.talendforge.org/browse/TUP-28934

Signed-off-by: jding-tlnd <jding@talend.com>
2020-11-11 10:45:23 +08:00
apoltavtsev
1df79674e1 feat(TESB-29949) Pass the data source to a job using a context variable 2020-11-10 15:57:31 +01:00
hcyi
c0fedf4ef4 feat(TUP-25235):tELTTeradataMap : allow to rename the alias. (#5491) 2020-11-10 11:59:47 +08:00
hcyi
f3b45bf229 feat(TUP-25235):tELTTeradataMap : allow to rename the alias. (#5284)
* feat(TUP-25235):tELTTeradataMap : allow to rename the alias.

* feat(TUP-25235):tELTTeradataMap : allow to rename the alias.

* feat(TUP-25235):improve and add some junits for rename alias.

* fix(TUP-25235):update some text messages for tELTTeradataMap : allow to
rename the alias

* fix(TUP-25235):update some text messages for tELTTeradataMap : allow to
rename the alias

* fix(TUP-25235):TUP-29225 [bug]ELTMAP should check alias name validation
.

* fix(TUP-25235):TUP-29225 [bug]ELTMAP should check alias name validation

* fix(TUP-25235):TUP-29225 [bug]ELTMAP should check alias name validation
2020-11-10 11:20:41 +08:00
wang wei
243a1f3326 fix(TDI-44993): tS3Get performance issue using multipart target to EFS/NAS(#5429) 2020-11-09 21:41:12 +08:00
wang wei
4abf245ca1 fix(TDI-44910): Add memsql support in studio (#5479) 2020-11-09 21:38:42 +08:00
Jane Ding
d71b4b148a Fix junit failure (#5489)
Signed-off-by: jding-tlnd <jding@talend.com>
2020-11-09 19:24:56 +08:00
Chao MENG
45f94c22be feat(TUP-28790): Enhance detection of localhost (#5379)
* feat(TUP-28790): Enhance detection of localhost

https://jira.talendforge.org/browse/TUP-28790

* feat(TUP-28790): Enhance detection of localhost

https://jira.talendforge.org/browse/TUP-28790
2020-11-09 17:31:55 +08:00
chmyga
1c41f0c05d chore(TDI-44004): update CXF and talend-ws version (#5485)
Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-11-09 10:34:51 +02:00
Chao MENG
d70ad09a49 feat(TUP-29126): Github Renaming the default branch from master (#5457)
* feat(TUP-29126): Github Renaming the default branch from master
https://jira.talendforge.org/browse/TUP-29126

* feat(TUP-29126): Github Renaming the default branch from master
https://jira.talendforge.org/browse/TUP-29126
2020-11-09 10:10:45 +08:00
bhe-talendbj
6e440ed726 fix(TUP-29164): Remove duplicated invokes of checkStartNodes (#5454) 2020-11-06 16:44:28 +08:00
kjwang
d8ace9d577 temp commit (#5411)
TUP-28833 Multi-User: support custom javajet component in shared studio
https://jira.talendforge.org/browse/TUP-28833
2020-11-06 15:02:22 +08:00
Jane Ding
8b3bbf7dcb feat(TUP-29076):support memsql in studio metadata and components (#5441)
* feat(TUP-29076):support memsql in studio metadata and components
https://jira.talendforge.org/browse/TUP-29076

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29076):support memsql in studio metadata and components
https://jira.talendforge.org/browse/TUP-29076
fix(TUP-29101):[bug] data viewer failed when export to context
https://jira.talendforge.org/browse/TUP-29101
Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-29076):support memsql in studio metadata and components
https://jira.talendforge.org/browse/TUP-29076

Signed-off-by: jding-tlnd <jding@talend.com>
2020-11-06 10:52:34 +08:00
pyzhou
0d0a67be0c Pyzhou/tdi 45062 t file copy last modified time 7.3 (#5472)
* fix(TDI-45062):add back last modification time

* add copy
2020-11-05 17:04:50 +08:00
clesaec
3e410b5373 TDI-44941 : ftp put connectors, renames (#5366)
* TDI-44941 : ftp put connectors, renames
2020-11-04 12:17:06 +01:00
pyzhou-talend
aad3651c38 fix(TDI-44924):change UI string 2020-11-04 16:04:59 +08:00
Colm O hEigeartaigh
6515a196e2 TDI-44685 - Close streams after calling getResourceAsStream in generated code (#5065) 2020-11-04 15:58:28 +08:00
mbasiuk-talend
cdc51a076c chore(TDI-44994): bump library version (#5427)
* chore(TDI-44994): bump library version

* chore(TDI-44994): change version to lower version, avoid bumping other libs

* chore(TDI-44994): update correct library
2020-11-03 09:49:45 +02:00
pyzhou
5e327d255d fix(TDI-44622):Correct Bonita mvn URL (#4992)
* fix(TDI-44622):Correct Bonita mvn URL

* revert default version
2020-11-03 14:47:28 +08:00
kjwang
63fffa6d6c Kjwang/fix tup 28050 avoid possible actions (#5283)
Fix TUP-28050 Multi-User: Avoid possible actions that are not supported on shared studio
https://jira.talendforge.org/browse/TUP-28050
2020-11-03 11:23:27 +08:00
vyu-talend
9a974a67c0 fix(TDI-45054):fix the protected file error. (#5428)
* fix(TDI-45054):fix the protected file error.

* fix(TDI-45054):update the excel jar in component.

* fix(TDI-45054):update lib version.
2020-11-03 11:17:29 +08:00
bhe-talendbj
2a8a6c074a bugfix(TUP-29131) Replace jar name by maven uri for spark jobs (#5444)
* fix(TUP-29131): Replace jar name by uri for spark job

* fix(TUP-29131): Replace jar name by maven uri for spark jobs
2020-11-02 15:11:42 +08:00
pyzhou
53435511fb fix(TDI-45098):tHttpRequest accept 2XX as success (#5442) 2020-11-02 13:53:21 +08:00
Dmytro Grygorenko
7840e1783e fix(TDI-45023): Update tJasper* components to use Apache POI 4.1.2 (#5426)
* fix(TDI-45023): Update libraries for tJasper* components.

* fix(TDI-45023): fix for HTML report generation.
2020-10-29 15:01:22 +02:00
Jane Ding
48c26b2d21 fix(TUP-28945):For Snowflake connection, the tDBConnection which is (#5407)
* fix(TUP-28945):For Snowflake connection, the tDBConnection which is
displayed in the joblet (in tDBInput) is not retained in the job from
where the joblet is invoked
https://jira.talendforge.org/browse/TUP-28945

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28945):For Snowflake connection, the tDBConnection which is
displayed in the joblet (in tDBInput) is not retained in the job from
where the joblet is invoked
https://jira.talendforge.org/browse/TUP-28945

Signed-off-by: jding-tlnd <jding@talend.com>
2020-10-29 20:59:28 +08:00
bhe-talendbj
8027965b4c fix(TUP-28978): Add migration (#5422)
* fix(TUP-28978): Add migration

* fix(TUP-28978): show jar for tELMap, show uri for routine/bean dependencies
2020-10-29 15:55:55 +08:00
qiongli
e9bad7dd32 fix(TDQ-18826)Set required to true for Dynamic jars (#5425) 2020-10-28 18:02:47 +08:00
Dmytro Grygorenko
593a717084 fix(TDI-44087): Update Xstream to 1.4.11.1 version (#5430) 2020-10-28 10:39:29 +02:00
cbadillo1603
fb884595dd fix(TBD-11405): there's no button to browse credential files for tHiveXXX components in DI job (#5333) 2020-10-28 09:10:25 +01:00
clesaec
756ba629ed TDI-44686 : potential NPE (#5164)
* TDI-44686 : potential NPE
2020-10-26 09:59:02 +01:00
Roman
aac496bf8f chore(TDI-44004): update talend-ws lib (#5415)
* chore(TDI-44004): update talend-ws lib

* chore(TDI-44004): update cxf for MicrosoftCRM components
2020-10-26 09:33:28 +02:00
Andrii Medvedenko
7ac98d48a3 fix(TBD-11378): tHiveInput in DI has authentication type for all distributions (#5313)
* fix(TBD-11378): wrong property fix

* an other part of ofix
2020-10-23 15:37:34 +03:00
Dmytro Grygorenko
8752a75abd fix(TDI-45015): tFileInputMail - slow attachments retrieval. (#5399)
* fix(TDI-45015): fix for low Input/OutputStreams performance.

* fix(TDI-45015): increase buffer size to align sizes of internal and external buffers.

* fix(TDI-45015): moved flush() out of cycle.
2020-10-23 11:24:33 +03:00
pyzhou
9e85699b85 fix(TDI-44924):tFileFetch should not trust all server by default (#5328)
* fix(TDI-44924):tFileFetch should not trust all server by default

* add migration task

* correct class name

* fix plugin.xml
2020-10-23 10:41:45 +08:00
bhe-talendbj
c09016f435 fix(TUP-28066): remove the lower jar based on gavct (#5409) 2020-10-22 14:17:38 +08:00
bhe-talendbj
028b6681c3 bugfix(TUP-29022) Fix detection warning popup (#5400) (#5401)
* fix(TUP-29022): fix change warning popup

* fix(TUP-2902): show changes warning popup for implicit context and jobsettings
2020-10-21 14:07:00 +08:00
bhe-talendbj
22fbf0ff0b fix(TUP-29022): fix jobsettings jdbc change detector (#5394) 2020-10-20 18:14:28 +08:00
Oleksandr Zhelezniak
ec74ac0f35 fix(TDI-44935): reuse custom query expression (#5349)
* make query expression idempotent in the generated code
* use instead of duplication of custom query expression a variable
2020-10-20 12:33:53 +03:00
Mike Yan
c5d44ab48b fix(TESB-30713): Beans folder is missing in routines.jar for route (#5391) 2020-10-20 15:43:04 +08:00
Chao MENG
1db9854428 fix(TUP-29012): If I open spark job before di job, on di job run tab I (#5387)
can see spark configuration tab

https://jira.talendforge.org/browse/TUP-29012
2020-10-20 11:31:51 +08:00
Jane Ding
5d1ab30b34 fix(TUP-28952):TOS: NPE when delete items to recycle bin (#5383)
https://jira.talendforge.org/browse/TUP-28952

Signed-off-by: jding-tlnd <jding@talend.com>
2020-10-20 10:15:34 +08:00
Dmytro Sylaiev
3502a8e79a fix(TDI-44896): Get nbline from stdout of gpload (#5304)
* fix(TDI-44896): Get nbline from stdout of gpload

* fix(TDI-44896): Fix the error message for failing get NBLine

* fix(TDI-44896): Fix codegen error

* fix(TDI-44896): Fix log printing 0 despite the real result

* fix(TDI-44897): Add exit code of GPLoad (#5307)
2020-10-19 18:22:46 +03:00
ypiel
f3b91a3cac feat(TDI-44950) : Support oauth 2.0 in mscrm 2016 on-premise (#5300)
* feat(TDI-44950) : Support oauth 2.0 in mscrm 2016 on-premise

* feat(TDI-44950) : some adjustement after meeting + qa

* feat(TDI-44950) : update talend-mscrm version
2020-10-19 14:50:45 +02:00
AlixMetivier
e1bceeea2d fix(TBD-11446): add tFileOutputDelimited to UpdateSeparatorAndEscapeForDatasetAPI migration task (#5374) 2020-10-19 11:24:23 +02:00
cbadillo1603
967a3d94ce fix(TBD-5167): tBigQueryInput project label should same as tBigQueryConfiguration (#5375) 2020-10-16 15:38:21 +02:00
bhe-talendbj
69a5234730 feat(TUP-28342):Save maven url for components with parameter MODULE_LIST (#5278)
* feat(TUP-25246): support custom maven uri for components and dbwizard

* feat(TUP-28342):Save maven url for components with parameter MODULE_LIST

* feat(TUP-28342): Save driver jar as mvnurl for implicit context.

* feat(TUP-25246): Fix necessary UI updates

* feat(TUP-25246): add migration to replace jar name by uri

* feat(TUP-25246): add migration

* feat(TUP-25246): generate poms after migration

* feat(TUP-25246): fix migrate driver jar path

* feat(TUP-25246): Fix context jar path and disable migration and show jar name instead of uri for modulelist

* feat(TUP-25246): fix stats and logs

* feat(TUP-25246): show jar name instead of uri for module table

* feat(TUP-25246): remove quotes

* feat(TUP-25246): fix parse maven uri from context

* feat(TUP-25246): use maven uri of component instead of driver jar path

* feat(TUP-25246): add workaround for components

Co-authored-by: Zhiwei Xue <zwxue@talend.com>
2020-10-16 16:22:59 +08:00
pyzhou
c8f4dd2e6d fix(TDI-44997): fix compile error (#5355) 2020-10-16 14:20:28 +08:00
mbasiuk-talend
a0b0366bcb fix(TDI-44893): update component, and localprovider (#5372)
* fix(TDI-44893): update component, and localprovider

* fix(TDI-44893): use latest soap library
2020-10-16 07:28:01 +03:00
Jane Ding
f24633190e feat(TUP-28640):Improve JDBC database support framework to load (#5368)
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640
2020-10-16 01:37:17 +08:00
jiezhang-tlnd
8e4e04c9cd feat(TUP-28758)Add a warning when we login to a project and if there is (#5295)
migration to do
2020-10-15 15:18:09 +08:00
Liu Xinquan
05a0815778 fix(TDQ-18784) add a new migration task (#5363) 2020-10-15 10:08:01 +08:00
Jane Ding
965c02c58d feat(TUP-28640):Improve JDBC database support framework to load (#5367)
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640
2020-10-15 08:48:14 +08:00
Emmanuel GALLOIS
5eb4100d98 feat(TCOMP-1757): support context in design time actions (#5220)
* feat(TCOMP-1757): support context in design time actions
* feat(TCOMP-1757): add context selection when multiple contexts
2020-10-14 16:53:38 +02:00
vdrokov
b0059fdb12 TESB-30427: Unable to utilize the Runtime datasource in DI Jobs with DB credentials blank (#5271)
Discover jndi source by different keys from the job note.
2020-10-14 16:30:12 +02:00
ypiel
061f3dc431 fix(TDI-44866) : update mscrm lib to 3.4-20200923 2020-10-14 15:40:21 +02:00
Chao MENG
e85afa939c fix(TUP-28612): [TCK] Guess Schema button run the subjob instead of (#5345)
calling discover schema method
https://jira.talendforge.org/browse/TUP-28612
2020-10-14 15:39:40 +08:00
jzhao
cb475f4e5e fix(TDI-45008):tFileOutputDelimited, 'Use OS line separator as row separator when CSV Row separator is set to CR, LF or CRLF' does not work properly (#5364) 2020-10-14 15:37:49 +08:00
Xilai Dai
a76b52f3b9 fix(TESB-30136) javax.activation.UnsupportedDataTypeException (#5329) 2020-10-14 10:21:20 +08:00
bhe-talendbj
4689d45132 fix(TUP-28659): migrate jobs because remove snapshot jars (#5294) 2020-10-14 09:45:15 +08:00
Jane Ding
9b5eccc67c feat(TUP-28640):Improve JDBC database support framework to load (#5287)
* feat(TUP-28640):Improve JDBC database support framework to load
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640

* feat(TUP-28640):Improve JDBC database support framework to load
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640

* feat(TUP-28640):Improve JDBC database support framework to load
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640
fix(TUP-28746):[Bug] after do "save the property to metadata" studio
will throw error logs.
cf: connection infos with quotes

* feat(TUP-28640):Improve JDBC database support framework to load
supported DBs and components automatically
https://jira.talendforge.org/browse/TUP-28640
2020-10-13 16:22:42 +08:00
Jane Ding
53d4b392bc fix(TUP-28618):[Bug] db type dont show Delta in impact page. (#5275)
https://jira.talendforge.org/browse/TUP-28618
2020-10-13 16:20:48 +08:00
Dmytro Grygorenko
a2510f5e2a fix(TDI-44957): tSSH timeouts use (#5303)
* fix(TDI-44957): add timeouts use and configuration.

* fix(TDI-44957): replicate old component's behavior (unlimited session if no timeout specified)

* fix(TDI-44957): set NIO2_READ_TIMEOUT to be infinite too.
2020-10-13 11:01:48 +03:00
Dmytro Grygorenko
539710015b fix(TDI-44839): update POSTGRESQL driver (#5273)
Co-authored-by: wwang-talend <wwang@talend.com>
2020-10-13 10:40:46 +03:00
Mike Yan
a398a56e95 Yyan/feat/tesb 29271 route debug 731 (#5351)
* fix(TESB-29271): Fix NPE for Spack job debug type

* fix(TESB-29271): Hide add breakpoint menu in routelet editor
2020-10-13 15:15:36 +08:00
hzhao-talendbj
ba647bde38 tup-28783 (#5331) 2020-10-13 14:57:05 +08:00
sbliu
3b37b58fd0 TUP-28778 Add 'if' judgment before type conversion. 2020-10-13 10:53:12 +08:00
kjwang
cc07722ebb kjwang/Feat_TUP-27762_new_version_of_ci (#5248)
kjwang/Feat_TUP-27762_new_version_of_ci
2020-10-13 10:50:05 +08:00
hzhao-talendbj
625792e472 tup 27356 (#5279)
* tup 27356

* modify code
2020-10-12 16:09:23 +08:00
mbasiuk-talend
b55e6c1c02 fix(TDI-44893): upgrade libraries, update code, remove old libs (#5229)
* fix(TDI-44893): upgrade libraries, update code, remove old libs

* fix(TDI-44893): update local provider pom with newest version
2020-10-12 10:33:23 +03:00
Andreas Mattes
b43b149ba4 TESB-30623 Ensure routine classes are compiled before the library is created. (#5322) 2020-10-10 10:43:53 +02:00
Mike Yan
918cf4eed5 fix(TESB-29271): Fix NPE for Spack job debug type (#5321) 2020-10-09 23:01:05 +08:00
Mike Yan
5fcaae4e48 fix(TESB-29271): Fix for Spack job debug type (#5320) 2020-10-09 17:48:22 +08:00
Mike Yan
41231a43b6 fix(TESB-29271): Fix for headless issue (#5319) 2020-10-09 10:47:47 +08:00
Mike Yan
03eeaac507 feat(TESB-29271): Route debuging feature (#5280)
* feat(TESB-29271):Initical update for route debugger

* feat(TESB-29271): conditional breakpoint component tab

* feat(TESB-29271): Route debuging feature

* fix(TESB-29271): Cumulative fixes and code improvement

* fix(TESB-29271): Added default view(DI)

* fix(TESB-29271): Code improvements by code review

* feat(TESB-29271): code updated by review
2020-10-07 21:50:35 +08:00
chmyga
ad7316b2ce Dchmyga/tdi 44868 tgsput huge files (#5230)
* fix(TDI-44868): tGSPut with huge files

* Add property to set part size to upload files in parts

* fix(TDI-44868): tGSPut with huge files

* add migration task

* fix(TDI-44868): tGSPut with huge files

* Fix PR comment

* fix(TDI-44868): tGSPut with huge files

* Fix overflow problem

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-10-05 17:41:49 +03:00
Dmytro Grygorenko
31e27f97e0 fix(TDI-44880): set ERROR_MESSAGE in components. (#5207) 2020-10-05 17:20:34 +03:00
Dmytro Grygorenko
09fd069243 fix(TDI-44883): set value of ERROR_MESSAGE (#5216) 2020-09-29 16:26:04 +03:00
Dmytro Sylaiev
f5ea29812e fix(TDI-44771): Fix error code for prejob in multithread run (#5208) 2020-09-28 09:33:05 +03:00
Jane Ding
5acbcd229e fix(TUP-26156)tCreateTable change DBType and Property Type not work (#4465) (#5276)
* fix(TUP-26156)tCreateTable: change "DBType" and "Property Type" not work
https://jira.talendforge.org/browse/TUP-26156

* fix(TUP-26156)tCreateTable: change "DBType" and "Property Type" not work
https://jira.talendforge.org/browse/TUP-26156

* fix(TUP-26156)tCreateTable change DBType and Property Type not work
https://jira.talendforge.org/browse/TUP-26156
2020-09-26 17:11:27 +08:00
Zhiwei Xue
203187c05f Revert "feat(TUP-28342):Save maven url for components with parameter MODULE_LIST"
This reverts commit 50e7057eb9.
2020-09-25 21:35:06 +08:00
Dmytro Grygorenko
39996404a7 fix(TDI-44568): Cumulative changes for POI update task (all-in-one) (#5269)
* fix(TDI-44568): cumulative commit for all changes

* fix(TDI-44568): Distribution plugin added

* fix(TDI-44568): all latest changes

* fix(TDI-44568): recent changes

* fix(TDI-44568): latest + build fix

* fix(TDI-44568): group name changed, versions updated

* fix(TDI-44568): classpath and manifest files updated
2020-09-25 14:32:39 +03:00
Zhiwei Xue
50e7057eb9 feat(TUP-28342):Save maven url for components with parameter MODULE_LIST 2020-09-25 17:45:08 +08:00
pyzhou
94c39f79ff fix(TDI-44911):tFileInputExcel has malposition header when enabling (#5244)
column
2020-09-25 14:10:51 +08:00
SunChaoqun
a1b85f19c1 Bugfix/maintenance/7.3/tesb 30423 (#5267)
* TESB-30423:Install Patch_20200925_R2020-09_v1-7.3.1 with TDI.license,
studio export job fail

* TESB-30423:Install Patch_20200925_R2020-09_v1-7.3.1 with TDI.license,
studio export job fail
2020-09-24 13:33:18 +08:00
SunChaoqun
66755538e8 TESB-30423:Install Patch_20200925_R2020-09_v1-7.3.1 with TDI.license, (#5259)
studio export job fail
2020-09-23 18:32:09 +08:00
Emmanuel GALLOIS
a2209548ae fix(TCOMP-1772): typo 2020-09-23 11:43:37 +02:00
Emmanuel GALLOIS
d8e604bdb0 fix(TCOMP-1772): fix MEMO_X type in configuration.javajet (#5257) 2020-09-23 09:41:20 +02:00
Jane Ding
e4194eea26 feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE (#5255)
* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654
2020-09-23 14:20:12 +08:00
Dmytro Grygorenko
3a9ea6fafd Revert "fix(TDI-44568): Update TalendExcel library. (#5001)" (#5254)
This reverts commit e8c2b70985.
2020-09-22 17:06:25 +03:00
Dmytro Grygorenko
d71fce6268 Revert "fix(TDI-44568): Update talendMsgMailUtil library (#5040)" (#5253)
This reverts commit 75fb043c1c.
2020-09-22 17:06:18 +03:00
Dmytro Grygorenko
11368d3910 Revert "fix(TDI-44568): Update simpleexcel library (#5041)" (#5252)
This reverts commit 9b70682a48.
2020-09-22 17:06:12 +03:00
Dmytro Grygorenko
d01477ab11 Revert "fix(TDI-44568): Update org.talend.libraries.excel plugin (#5042)" (#5251)
This reverts commit e0b54b7df4.
2020-09-22 17:06:03 +03:00
Dmytro Grygorenko
7900e44e13 Revert "fix(TDI-44568): Update Apache POI version in components (#5043)" (#5250)
This reverts commit 04988ed52b.
2020-09-22 17:05:50 +03:00
Dmytro Grygorenko
04988ed52b fix(TDI-44568): Update Apache POI version in components (#5043)
* fix(TDI-44568): Update Apache POI version in components

* fix(TDI-44568): use custom POI libraries in components

* fix(TDI-44568): "simpleexcel" version updated.

* fix(TDI-44568): more complete version update )

* fix(TDI-44568): Jasper components to use previous POI version (could not be updated that easy).
2020-09-22 13:28:11 +03:00
Dmytro Grygorenko
e0b54b7df4 fix(TDI-44568): Update org.talend.libraries.excel plugin (#5042)
* fix(TDI-44568): Talend libraries versions updated.

* fix(TDI-44568): remove unnecessary files.

* fix(TDI-44568): classpath file restored

* fix(TDI-44568): use custom POI libraries

* fix(TDI-44568): Manifest file restored, versions updated.
2020-09-22 13:27:48 +03:00
Dmytro Grygorenko
9b70682a48 fix(TDI-44568): Update simpleexcel library (#5041)
* fix(TDI-44568): Update simpleexcel library

* fix(TDI-44568): Aligning code with "master" branch

* fix(TDI-44568): version updated

* fix(TDI-44568): code alignment reverted, library version changed.
2020-09-22 13:27:32 +03:00
Dmytro Grygorenko
75fb043c1c fix(TDI-44568): Update talendMsgMailUtil library (#5040) 2020-09-22 13:26:16 +03:00
Dmytro Grygorenko
e8c2b70985 fix(TDI-44568): Update TalendExcel library. (#5001)
* fix(TDI-44568): Bump up versions, remove unused dependencies, some refactoring.

* fix(TDI-44568): addressed issue with preWb.

* fix(TDI-44568): GAV update, removed "version = 6.0.0"

* fix(TDI-44568): Version updated
2020-09-22 13:25:56 +03:00
wang wei
e727191389 fix(TDI-44908): pass TMC function jar for trunjob independent process case and dynamic job case (#5235) 2020-09-22 18:14:48 +08:00
Jane Ding
340b59a17e feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE (#5240)
https://jira.talendforge.org/browse/TUP-27654
2020-09-21 19:59:33 +08:00
OleksiiNimych
6a2257f107 feat(TDI-39689): tTeradataTPT support for multiple schemas (#5193)
* feat(TDI-39689): tTeradataTPT support for multiple schemas

* feat(TDI-39689): tTeradataTPTUtility

* feat(TDI-39689): tTeradataTPT add migration task

* feat(TDI-39689): rename new property

* feat(TDI-39689): tTeradataTPT fix migration task
2020-09-21 11:36:14 +03:00
Jane Ding
d4751f4cfb feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE (#5233)
https://jira.talendforge.org/browse/TUP-27654
2020-09-21 14:39:27 +08:00
hcyi
9e9406e2fa fix(TUP-25171):Issue when using components inside Joblet. (#5119)
* fix(TUP-25171):Issue when using components inside Joblet.

* fix(TUP-25171):Issue when using components inside Joblet.

* fix(TUP-25171):code format for Issue when using components inside
Joblet.
2020-09-18 15:55:09 +08:00
Chao MENG
647dbe0676 fix(TUP-26413): Be able to compare metadata connection conflicts (#5222)
https://jira.talendforge.org/browse/TUP-26413
2020-09-18 12:25:18 +08:00
Jane Ding
775e5a59f6 feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE (#5078)
* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

Signed-off-by: jding-tlnd <jding@talend.com>

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654
TUP-28598:[Bug] driver class show as mvn:SparkJDBC42-2.6.14.1018.jar is
wrong.

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
TUP-28615:[Bug] Save the property to metadata will change Delta Lake to
JDBC.
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
TUP-28610:[Bug] drag Delta Lake component from Palatte to job meet NPE.
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654
Hard code to filter unsupported components
Set useAutoCommit and autocommit default value as true

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
TUP-28616:[Bug] in stat&logs/extra page, when select Delta Lake DB, it
shows as JDBC.
To fit on support database, delta lake should be filter
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
set delta lake/jdbc default mapping
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654

* feat(TUP-27654):Databricks Delta Lake - Support ELT & MERGE
https://jira.talendforge.org/browse/TUP-27654
2020-09-18 12:01:14 +08:00
wang wei
9a18f18a76 fix(TDI-44745): FileInputDelimited fails to properly consume CipherInputStream(#5140) 2020-09-17 14:15:43 +08:00
Chao MENG
7a66f2715c Cmeng/patch/7.3/tup 28063 improve start (#5092)
* workitem(TUP-28063): Improve studio bundles reloading logic
https://jira.talendforge.org/browse/TUP-28063

* workitem(TUP-28063): Improve studio bundles reloading logic
https://jira.talendforge.org/browse/TUP-28063

* workitem(TUP-28063): Improve studio bundles reloading logic
https://jira.talendforge.org/browse/TUP-28063
2020-09-16 16:40:47 +08:00
Zhiwei Xue
33c569d7cb feat(TUP-28323):Improve the maven build time spent during login project (#5166) 2020-09-16 16:35:24 +08:00
bhe-talendbj
41d611a140 fix(TUP-28628): NPE because of context (#5206) 2020-09-16 15:46:28 +08:00
vdrokov
7381677f63 fix(TESB-29853): TESB-29819 Studio doesn't select model beans during exporting (#5002)
* fix(TESB-29819): Studio doesn't select model beans during exporting

* fix(TESB-29819) Studio doesn't select model beans during exporting
Alternative fix
2020-09-16 08:19:49 +02:00
pyzhou
0685203687 fix(TDI-44855):fix index out of bound error (#5212) 2020-09-16 12:26:04 +08:00
wang wei
02bd1c98e7 fix(TDI-44824): Observability: Update daikon-audit library version (#5205) 2020-09-15 20:26:41 +08:00
apoltavtsev
2115adb42e fix(TESB-29963) MQ Dependency issues in Runtime (#5104)
* fix(TESB-29963) MQ Dependency issues in Runtime

* Update JobJavaScriptOSGIForESBManager.java
2020-09-15 11:38:27 +02:00
wang wei
cfd4a491ea fix(TDI-44806): Observability: Add customization for event frequency (#5200)
* fix(TDI-44806): Observability: Add customization for event frequency

* adjust the key to "audit.interval"
2020-09-15 17:30:15 +08:00
hcyi
c1c0c5481b fix(TUP-28635):[bug] output path always be changed when path contains \ (#5209) 2020-09-15 14:51:49 +08:00
Zhiwei Xue
9b6eb4d7ab fix(TUP-28576):Reference projects routine POM file is modified and can't (#5204)
be committed to GIT
2020-09-15 09:56:07 +08:00
hcyi
ebb4502761 feat(TUP-23827):Runtime Lineage for studio. (#5003)
* feat(TUP-23827):Runtime Lineage for studio.

* feat(TUP-23827):improve the GUI of Runtime Lineage for studio.

* feat(TUP-23827):improve the GUI of Runtime Lineage for studio.

* feat(TUP-23827):TUP-28186[bug]runtimelineage.prefs better remove job
version

* feat(TUP-23827):update the text Runtime Lineage to lower case after
review

* fix(TDI-44670): Runtime Lineage - Finalize component part

* split audit log and runtime lineage log

* add a custom api for other feature and future

* feat(TUP-23827):only EE product can have runtime lineage。

* fix something

* fix trunjob

* fix(TDI-44735): remove the duplicated parameters info in message, user
can fetch it from customInfo json object

* fix(TDI-44734): Enhance output path system for logging of runtime json

* fix the compiler issue

* filter the special table field

* fix the compiler and NPE risk

* feat(TUP-23827):only support for DI jobs if selected "Use runtime
lineage for all jobs" .

* feat(TUP-28539):Enhance GUI of Runtime Lineage.

* feat(TUP-28539):improve for Enhance GUI of Runtime Lineage.

* fix the compiler issue

* feat(TUP-28539):remove Output path test and verify 。

Co-authored-by: wwang-talend <wwang@talend.com>
2020-09-14 18:42:55 +08:00
Emmanuel GALLOIS
38ce7af3e9 fix(TCOMP-1770): bump component-runtime to 1.1.25 2020-09-14 11:10:26 +02:00
pyzhou
e5c07ac253 fix(TDI-44855):tSSH append new line in the result (#5190) 2020-09-14 10:58:32 +08:00
AlixMetivier
176e3489ce fix(TBD-10885): refacto date and timestamp handling with dataset (#5149) 2020-09-10 14:48:45 +02:00
vyu-talend
65c980efdf fix(TDI-44846):fix the compliation error. (#5187) 2020-09-10 14:04:58 +08:00
nrousseau
2fb8abe293 fix(TUP-28487): fix performance issue (#5152)
* fix(TUP-28487): fix performance issue
2020-09-10 09:52:59 +08:00
vyu-talend
036736f589 fix(TDI-44803):attachment issue in tfileinputmail. (#5176) 2020-09-09 18:08:02 +08:00
OleksiiNimych
5b8b168860 fix(TDI-44759): tTeradataTPTExec fix SHOW_IF statement (#5147)
* fix(TDI-44759): tTeradataTPTExec fix SHOW_IF statement

* fix(TDI-44759): tTeradataTPTExec fix display of properties
2020-09-09 10:20:36 +03:00
Oleksandr Zhelezniak
ea82a11f32 chore(TDI-44765): remove guava:13.0 (#5168)
* remove guava:13.0 dependency use 20.0 instead of it
2020-09-09 09:58:49 +03:00
chmyga
81e7e49dcf fix(TCOMP-1749): validation with ActiveIf broken (#5121)
* Don't show validation results for hidden parameters

* Revalidate parameters on show/hide property change

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-09-08 15:58:28 +03:00
Emmanuel GALLOIS
e5f2b94511 feat(TCOMP-1651): handle Dynamic for component-runtime components (#4861)
* feat(TCOMP-1651): bump component-runtime to 1.1.23-SNAPSHOT
- should be changed after the 1.1.23 release
* feat(TCOMP-1651): javajet update to Dynamic
* feat(TCOMP-1651): copyright update
* feat(TCOMP-1651): update javajet stubs
* feat(TCOMP-1651): update configuration javajet
* feat(TCOMP-1651): fix errors on configuration javajet
* feat(TCOMP-1651): bump component-runtime to 1.1.23
* feat(TCOMP-1651): bump component-runtime to 1.1.24-SNAPSHOT
* feat(TCOMP-1651): pass dynamic column metadatas to RecordConverters
* feat(TCOMP-1651): bump component-runtime to 1.1.24
2020-09-08 10:19:37 +02:00
Andrii Medvedenko
1b6f6cb3cb feat(TBD-10473): Hive input components for Dataproc distribution to work with Service account (#5114) 2020-09-08 02:34:40 +03:00
wang wei
e7a58886dc fix(TDI-44760): TMomOutput not changing the value of the queue (#5145) 2020-09-07 16:59:17 +08:00
wang wei
84d87eb37a fix(TDI-44678): tFileOutputXML make unexpected empty file (#5052) 2020-09-07 15:37:20 +08:00
vyu-talend
3506e3b76a feat(TDI-44720):implement pause and resume for redshift cluster. (#5141)
* feat(TDI-44720):implement pause and resume for redshift cluster.

* feat(TDI-44720):change show-if.
2020-09-04 14:28:02 +08:00
vyu-talend
60b1216d84 feat(TDI-44755):upgrade aws sdk version. (#5139) 2020-09-04 13:55:49 +08:00
hcyi
38635cfa1d fix(TUP-28258):tETLInput and tETLMap does not show datatypes for (#5124)
Snowflake DB on Import Metadata
2020-09-04 11:28:09 +08:00
Emmanuel GALLOIS
73c93317fd (TCOMP-1759): fix random order for columns in guessSchema in Studio (#5151) 2020-09-03 11:36:46 +02:00
Oleksandr Zhelezniak
584dd71b8b fix(TDI-44654): fixed NPE for tFTP (#5071)
* add info message about nullity of the ssl session context
* update version of commons-net-ftps-proxy [3.6.1-talend-20190819] -> [3.6.1-talend-20200902]
* update version of 'commons-net-ftps-proxy' in javajet ftp's components
* remove slf4j-log4j12 dependency
2020-09-03 10:17:24 +03:00
jzhao
7179453025 fix(TDI-30522): revert all fixes. (#5163)
* Revert "fix(TDI-30522): extra '(' removed (#5161)"

This reverts commit 9497eee89e.

* Revert "fix(TDI-30522): reworked fix. (#5109)"

This reverts commit 478a707a7c.
2020-09-02 15:00:08 +08:00
Dmytro Grygorenko
9497eee89e fix(TDI-30522): extra '(' removed (#5161)
* fix(TDI-30522): extra '(' removed

* fix(TDI-30522): missing '<%' restored
2020-09-01 20:42:10 +03:00
Dmytro Grygorenko
b8bae80459 TalendExcel: read password protected files. (#5132)
* fix(TDI-44743): read password protected files.

* fix(TDI-44743): use new library.

* fix(TDI-44743): Error message changed.
2020-09-01 16:11:08 +03:00
Oleksandr Zhelezniak
8827109f7f chore(TDI-44145): update commons-codec:1.14 (#5154)
* update commons-codec to 1.14 in javajets
* update internal libraries(libs_src) where was updated commons-codec
2020-09-01 15:26:43 +03:00
jzhao
a2ffcbba3e fix(TDI-44748):Error in method "add" for the component tFileOutputMSXML (#5133) 2020-09-01 15:07:13 +08:00
mbasiuk-talend
6c5c7bcfec feat(TDI-44730): upgrade snowflake driver to latest (#5150) 2020-09-01 09:52:41 +03:00
Dmytro Sylaiev
c5f8b26687 fix(TDI-44074): Fix performance for date mssql (#4638) (#5155) 2020-08-31 17:32:58 +03:00
Oleksandr Zhelezniak
614556ce8c fix(TDI-31162): remove interfere condition (#5120)
* remove interfere SPECIFY_DATASOURCE_ALIAS from the import condition of the tOracle components
2020-08-31 10:42:48 +03:00
Dmytro Grygorenko
478a707a7c fix(TDI-30522): reworked fix. (#5109)
* fix(TDI-30522): reworked fix.

* fix(TDI-30522): Closing bracket restored
2020-08-28 21:03:49 +03:00
Jane Ding
a7c9258846 Jding/tup 28440 (#5138)
* Add test suffix for test case support (#5136)

Co-authored-by: Michael Verrilli <mverrilli@talend.com>

* Mverrilli patch 7.3.1 iterate test fix (#5137)

* Add test suffix for test case support

* Fixed missing reference to className

* Update iterate_subprocess_header.javajet

Co-authored-by: Michael Verrilli <mverrilli@talend.com>

Co-authored-by: Michael Verrilli <mverrilli@talend.com>
2020-08-28 15:03:11 +08:00
Jane Ding
4f63e93995 Jding/73/tup 28085 error while compiling a kafka component with big data studio (#5089)
* fix(TUP-28085):Error while compiling a Kafka component with Big Data
studio
https://jira.talendforge.org/browse/TUP-28085
Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28085):Error while compiling a Kafka component with Big Data
studio
https://jira.talendforge.org/browse/TUP-28085

Signed-off-by: jding-tlnd <jding@talend.com>
2020-08-28 14:01:41 +08:00
Emmanuel GALLOIS
9a44a3006f patch(TPS-4101): bump component-runtime to 1.1.15.2 (#5127) 2020-08-25 12:42:02 +02:00
Roman
59da4a3e14 fix(TDI-44063): add transitive xml-apis lib (#5128) 2020-08-25 11:48:42 +03:00
Dmytro Sylaiev
27928d664f fix(TDI-44394): Fix migration tcomp for javajet (#5129) 2020-08-25 11:07:09 +03:00
bhe-talendbj
e2bc6699fa fix(TUP-28308): build error and joblet maven xml file has incorrect values and Couldn't parse data error message (#5116)
* fix(TUP-28308): Check multiple version of joblets

* fix(TUP-28308): Revert unnecessary changes
2020-08-25 10:27:22 +08:00
hcyi
d840d36182 fix(TUP-28134):TCK Components shown as hidden by default in the palette (#5045)
settings
2020-08-25 09:57:02 +08:00
vdrokov
1492cf8130 fix(TESB-29980): Issue of tsendemail component deployed in runtime wi… (#5108)
* fix(TESB-29980): Issue of tsendemail component deployed in runtime with attachment

* TESB-29980 Simplified fix as it should cover Java 8 and 11.

Co-authored-by: Andreas Mattes <andreasmattes@operamail.com>
2020-08-24 19:00:21 +02:00
jiezhang-tlnd
bce81ea615 fix(TUP-28078)metadata folder generated under .../poms/jobs when create (#5075)
new connection in remote project
https://jira.talendforge.org/browse/TUP-28078
2020-08-24 10:28:58 +08:00
Emmanuel GALLOIS
c49a50b971 fix(TDI-44164): revert changes on talend.component.manager.m2.repository (#5111) 2020-08-21 17:19:39 +02:00
Roman
1f66234efa fix(TDI-44063): update xercesImpl dependency (#5004)
* fix(TDI-44063): update xercesImpl dependency (#4857)

* fix(TDI-44063): bump talend soap lib in maven dependency plugin

* fix(TDI-44063): back bundleID

* fix(TDI-44063): change GAV for soap lib
2020-08-21 17:46:20 +03:00
Dmytro Grygorenko
fd1fd16679 fix(TDI-44439): backport to "maintenance/7.3" (#5029)
* fix(TDI-44439): backport to "maintenance/7.3"

* fix(TDI-44439): in case dbName is null
2020-08-21 13:14:05 +03:00
Jane Ding
c7213657a5 fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes (#5101)
* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

Signed-off-by: jding-tlnd <jding@talend.com>

* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

Signed-off-by: jding-tlnd <jding@talend.com>
2020-08-20 18:35:55 +08:00
Pierre Teyssier
d362662fa1 fix(TDI-42668): replace hardcoded value 2020-08-20 04:27:31 +08:00
Andrii Medvedenko
5d53d880ed fix of tHiveInput (#5100)
tHiveInput with DataProc 1.4 in DI: no rows retrieved
2020-08-19 17:02:30 +03:00
Jane Ding
1d508903ea Jding/tup 28338 implicit context issue (#5097)
* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338

* fix(TUP-28338):[7.3.1] configuring Implicit tContextLoad in the job Goes
on error after installing the patch 07_2020
https://jira.talendforge.org/browse/TUP-28338
2020-08-19 18:49:00 +08:00
Mike Yan
87c312f5b5 fix(TESB-29957): Added properties for metrics (#5095)
* fix(TESB-29957):CI Fail to publish Microservice with Prometheus param

* fix(TESB-29957): Added contexts properties

* fix(TESB-29957): Added properties for metrics
2020-08-19 17:05:21 +08:00
Chao MENG
f206247506 fix(TUP-28295): UI Problem of TCOMPV0 in the recent 741NB and 731 (#5079)
Release + Aug. Nightly Build Patch
https://jira.talendforge.org/browse/TUP-28295
2020-08-17 15:43:56 +08:00
Mike Yan
c8ee01160c fix(TESB-29957): Added contexts properties (#5082)
* fix(TESB-29957):CI Fail to publish Microservice with Prometheus param

* fix(TESB-29957): Added contexts properties
2020-08-14 17:33:45 +08:00
Dmytro Grygorenko
8dd10f9d2d Update libraries: "httpclient-4.5.12" and "httpcore-4.4.13" (#4889)
* fix(TDI-44486): library update - "httpclient-4.5.12.jar"

* fix(TDI-44486): Updated httpcore and httpclient for all components.

* fix(TDI-44486): recent changes + "commons-codec-1.14.jar"
2020-08-14 11:53:53 +03:00
vyu-talend
fb83719828 feat(TDI-44630):support amazon EMR HA. (#5059)
* feat(TDI-44630):support amazon EMR HA.

* feat(TDI-44630):improve the error message.
2020-08-14 15:55:57 +08:00
Dmytro Sylaiev
3fa500b3ac tS3Connection setPathStyleAccess for better MinIO support (#5012) (#5076)
* Added a checkbox under Advanced settings of tS3Connection when a custom endpoint is used.
When MinIO is used via DNS/hostname this must be checked otherwise the operations fail.

* PathStyleAccess should only be used if a custom endpoint is enabled.

Co-authored-by: bgunics-talend <63251373+bgunics-talend@users.noreply.github.com>
2020-08-13 13:53:28 +03:00
Mike Yan
219c78267e fix(TESB-29957):CI Fail to publish Microservice with Prometheus param (#5073) 2020-08-13 17:38:54 +08:00
AlixMetivier
f43e648cd8 feat(TBD-10769): delta Features for R08 (#5067) 2020-08-13 10:16:58 +02:00
hzhao-talendbj
c69c81bba7 Hengzhao/backport73/tup 25103 (#5033)
* bugfix(TUP-25103):tELTMap - ctrl+space shortcut is not working to
retrieve easily job context

* TUP-25103

* Tup-25103

* tup-25103  use styleText

* tup-25103 add exception handler

* remove comments

* solve issues

* add doit false to avoid new line

* try replace "\r\n" to " " in tELTMssqlMap_main.javajet

* fix edit expression builder but sql tab not refresh

* remove comments and format

* improve code

* to fix other eltmap about multi-line expression  and import/export
expression from file

* remove javajet and remove buttons

* remove expressionbuilder

Co-authored-by: hwang <hwang@talend.com>
2020-08-13 13:56:20 +08:00
pyzhou
82a927aed4 feat(TDI-42668):update jar of tssh component branch 7.3 (#5038)
* feat(TDI-42668):update jar of tssh component

* fix status code

* fix error interactive

* support dsa

* fix compile error
2020-08-13 12:25:37 +08:00
Jane Ding
b507c601f2 fix(TUP-28180):[bug]Only latest version deleted from root pom when check (#5044)
"Exclude deleted items" and delete to recycle bin
https://jira.talendforge.org/browse/TUP-28180

Signed-off-by: jding-tlnd <jding@talend.com>
2020-08-13 10:42:52 +08:00
SunChaoqun
b7c7689554 Csun/tesb 29322/microservice monitoring expose camel metrics m73 (#5060)
Merge into maintenance/7.3 for patch  7.3.1-R2020-08 - 20200821, since master branch has been validated.
2020-08-13 10:42:06 +08:00
Dmytro Grygorenko
1dce8ebdb8 Revert "fix(TDI-30522): backport to "maintenance/7.3" branch (#4898)" (#5070)
This reverts commit 4380916dcf.
2020-08-12 11:20:05 +03:00
bhe-talendbj
dfb1157d92 fix(TUP-27374): Fix test case (#5055) (#5061) 2020-08-11 14:53:32 +08:00
SunChaoqun
3ee0dd2559 TESB-29133 Update dom4j to 2.1.3 (#5000) (#5053)
Co-authored-by: Maksym Sheverda <maksym.sheverda@synapse.com>

Co-authored-by: msheverda-talend <53863455+msheverda-talend@users.noreply.github.com>
Co-authored-by: Maksym Sheverda <maksym.sheverda@synapse.com>
2020-08-11 13:56:33 +08:00
wang wei
225ce06412 fix(TDI-44192): Update dom4j to 2.1.3 (#4899) 2020-08-11 10:21:07 +08:00
AlixMetivier
0a18ad8c80 fix(TBD-11052): fix dataset migration for csv value (#5051)
* fix(TBD-11052): wrong value was checked

* fix value
2020-08-10 09:39:02 +02:00
jiezhang-tlnd
b3fca54bde fix(TUP-27697)Studio throw NPE error when drag json item to job (#4967)
https://jira.talendforge.org/browse/TUP-27697
2020-08-10 10:58:25 +08:00
AlixMetivier
47b3c48580 fix(TBD-11052): add migration task for dataset API in delimited compo… (#5036) 2020-08-07 15:10:19 +02:00
wang wei
b14945f092 fix(TDI-44481): upgrade commons-beanutils to 1.9.4 (#4893) 2020-08-06 12:45:16 +08:00
bhe-talendbj
823443b922 bugfix(TUP-27374) Commandline doesn''t pick up the correct JVM arguments (#5028)
* fix(TUP-27374): Get default jvm arguments

* fix(TUP-27374): add back default vm arguments
2020-08-06 11:03:58 +08:00
Roman
6c7c7e2d0d fix(TDI-44658): rearrange parameters in UI for BigQueryBulkExec (#5035) 2020-08-05 09:54:26 +03:00
Oleksandr Zhelezniak
a05b884249 chore(TDI-44362): CVE. Remove commons logging (#5027)
* remove commons-logging library from mdm_transaction and webservice-input libraries
* bump version for talend-webservice-input [1.0.0-20170301]->[1.0.2]
* declare explicit version of maven-jar-plugin in talend-webservice-input dependency
* change group id of mdm_transaction
* update version of mdm_transaction [1]->[1.0.1]
* change import in javajet xml due to convension for the talend-webservice-input to [1.0.2/6.0.0]
2020-08-05 09:16:03 +03:00
Roman
a36c14163a chore(TDI-44551): rename parameter (#5030) 2020-08-04 17:38:10 +03:00
jiezhang-tlnd
efe395f1bb feat(TUP-27862)Add possibility to customize pattern for dates when (#4977)
retrieve table
2020-08-03 09:41:16 +08:00
wang wei
6353bf261a fix(TDI-44643): NPE exception on tWaitForSQLData with snowflake connection(#5006) 2020-07-31 11:23:59 +08:00
Chao MENG
441e39a51e fix(TUP-27953): tSalesforceInput component displays in-job (#4997)
tSalesforceConnection incorrectly
https://jira.talendforge.org/browse/TUP-27953

Conflicts:
	main/plugins/org.talend.designer.core.generic/src/main/java/org/talend/designer/core/generic/controller/ComponentRefController.java
2020-07-31 11:11:23 +08:00
Jane Ding
c803692a65 feat(TUP-25566)All the recycle bin should not be build or executed (#4961)
* feat(TUP-25566)All the recycle bin should not be build or executed
https://jira.talendforge.org/browse/TUP-25566

* feat(TUP-25566)All the recycle bin should not be build or executed
https://jira.talendforge.org/browse/TUP-25566
2020-07-31 02:35:41 +08:00
Dmytro Sylaiev
a2bef6021d fix(TDI-44512): Bump mscrm lib version for localprovider components (#4973) 2020-07-30 10:40:01 +03:00
sbliu
f6fe7cc804 TUP-27812 By upgrading asciidoctorj to 2.3.0 to upgrade jruby-stdlib then upgrade bctls-jdk15on to 1.62 2020-07-29 15:09:06 +08:00
bhe-talendbj
8c167c470d fix(TUP-28009): Save relationship index after version change (#4988) 2020-07-28 16:12:30 +08:00
kjwang
9c91e6cce0 Kjwang/fix tup 27879 library data.index (#4935)
Fix:TUP-27879 Shared user can't write library index file library_data.index
https://jira.talendforge.org/browse/TUP-27879
Improve the code fix a spelling errors
2020-07-28 09:16:09 +08:00
Dmytro Sylaiev
a003ea6a61 fix(TDI-44579): Hide encrypt button for parquet (#4979)
* fix(TDI-44579): Hide encrypt button for parquet

* fix(TDI-44579): Don't generate mskey when encrypt is hidden
2020-07-24 10:51:30 +03:00
hzhao-talendbj
dd6a40cd68 tup-27447 backport to 73 (#4823)
* tup-27447 backport to 73

* TUP-27447
fix duplicated jars in .bat
2020-07-24 11:13:45 +08:00
msheverda-talend
a67032cdfa TESB-29285, TESB-29284 Error occurred when execute a microservice jar which is built by a tresrequest job with embedded trestclient job (#4781)
* TESB-29285 Error occurred when execute a microservice jar which is built by a tresrequest job with embedded trestclient job

* TESB-29285 Error occurred when execute a microservice jar which is built by a tresrequest job with embedded trestclient job

* TESB-29285 Error occurred when execute a microservice jar which is built by a tresrequest job with embedded trestclient job

Co-authored-by: Maksym Sheverda <maksym.sheverda@synapse.com>
2020-07-23 11:32:01 +02:00
OleksiiNimych
2a38cf7a5b fix(TDI-44400): update google-cloud-bigquery (#4860) 2020-07-23 10:22:52 +03:00
Dmytro Grygorenko
4380916dcf fix(TDI-30522): backport to "maintenance/7.3" branch (#4898)
* fix(TDI-30522): backport to "maintenance/7.3" branch

* fix(TDI-30522): Fixed compile error for tMSSqlOutput component.

* fix(TDI-30522): Some more compile errors fixed.
2020-07-23 10:16:50 +03:00
Jane Ding
7c7e1efb24 fix json xpath issue (#4980) 2020-07-22 17:53:57 +08:00
jiezhang-tlnd
fc5dca5f67 fix(TUP-27588)tCreateTable data types mapping in the schema editor is (#4968)
always Mysql after dragging and dropping from database Metadata
2020-07-21 11:31:48 +08:00
apoltavtsev
54c951c4bd TESB-29647 Bundle version for Jobs is corrected 2020-07-20 13:44:37 +02:00
Richard Lecomte
320c282a09 TDI-44448 : AWS EMR version 5.29.0 (#4914)
* TDI-44448 : AWS EMR version 5.29.0

Co-authored-by: Pierre Teyssier <pteyssier@talend.com>
2020-07-17 15:38:25 +08:00
Jane Ding
d9bff92a85 fix(TUP-27179):Studio freezes on job importing/loading (#4682) (#4875)
* fix(TUP-27179):Studio freezes on job importing/loading
https://jira.talendforge.org/browse/TUP-27179

* fix(TUP-27179):Studio freezes on job importing/loading
https://jira.talendforge.org/browse/TUP-27179

* fix(TUP-27179):Studio freezes on job importing/loading
https://jira.talendforge.org/browse/TUP-27179
2020-07-17 11:51:54 +08:00
msheverda-talend
a360ffd1a7 TESB-29664 Reverting changes (#4954)
Co-authored-by: Maksym Sheverda <maksym.sheverda@synapse.com>
2020-07-16 18:07:28 +03:00
Roman
3d7aa5a71b fix(TDI-44562): replace all backslashes in GS path (#4949) 2020-07-16 10:52:13 +03:00
Roman
1bd53e9905 fix(TDI-44561): fix compile error when dieOnError is checked (#4944) 2020-07-16 08:04:55 +03:00
Dmytro Sylaiev
5580b4f8d2 feat(TDI-44476): Add parquet support for tReshiftUnload (#4908)
* feat(TDI-44476): Add parquet support for tReshiftUnload

* feat(TDI-44476): Hide comress property for parquet
2020-07-15 18:20:14 +03:00
Roman
9cbf21f766 feat(TDI-44466): add service account support for BulkExec and Output (#4909)
* feat(TDI-44466): add properties for service account

* feat(TDI-44466): add service account for BulkExec

* feat(TDI-44466): add service account for Output

* feat(TDI-44466): marked old auth type as deprecated

* feat(TDI-44466): rename param

* feat(TDI-44466): add migration task

* feat(TDI-44466): change default auth type

* feat(TDI-44466): rollback imports

* feat(TDI-44466): align dependencies

* feat(TDI-44466): hide auth type field when bulkfile exist
2020-07-15 14:24:52 +03:00
Roman
4f35e05d28 feat(TDI-38246): add new auth type to google storage family (align deps) (#4924)
* feat(TDI-38246): add new auth type to tGSConnection

* feat(TDI-38246): add auth to UI | add dependencies

* feat(TDI-38246): add new auth type to tGSBucketCreate

* feat(TDI-38246): add new auth type to tGSBucketDelete | small fixes

* feat(TDI-38246): add new auth type to tGSBucketExist

* feat(TDI-38246): add new auth type to tGSBucketList

* feat(TDI-38246): remove projectId from tGSBucketList

* feat(TDI-38246): add new auth type to tGSClose

* feat(TDI-38246): add filemap generation for blob

* feat(TDI-38246): add new auth type to tGSPut

* feat(TDI-38246): add new auth type to tGSDelete

* feat(TDI-38246): add new auth type to tGSList

* feat(TDI-38246): add new auth type to tGSGet

* feat(TDI-38246): add new auth type to tGSCopy

* feat(TDI-38246): remove projectID for service account auth type

* feat(TDI-38246): rename component properties

* feat(TDI-38246): extend locations for bucket creation

* feat(TDI-38246): remove dual locations

* feat(TDI-38246): change default auth type

* feat(TDI-38246): add migration task

* feat(TDI-38246): remove unused var

* feat(TDI-38246): improve migration task

* feat(TDI-38246): clean code

* feat(TDI-38246): correct showing some params when use exist connection

* feat(TDI-38246): mark all HMAC keys as deprecated

* feat(TDI-38246): avoid blob directories

* feat(TDI-38246): allign dependencies

* feat(TDI-38246): remove duplicate

* feat(TDI-38246): bump GCS lib version

* feat(TDI-38246): bump google api service storage

* feat(TDI-38246): update required deps

* feat(TDI-38246): align depepndencies for BigQuery family
2020-07-15 14:23:41 +03:00
wang wei
c3268d3afa fix(TDI-44482): CVE: org.apache.ant:ant:1.10.1 (#4902) 2020-07-15 18:01:48 +08:00
OleksiiNimych
f5c24520ca fix(TDI-44202): tCreateTable add support of Sybase (#4780) 2020-07-15 15:41:29 +08:00
pyzhou
0e30f970cf fix(TDI-44558):fix tJDBCSCDELT 65535 issue (#4938) 2020-07-15 15:24:58 +08:00
jzhao
8782a4840f fix(TDI-44453):tAdvancedFileOutputXML issue after upgrading to 7.3.1 (#4927) 2020-07-15 14:02:57 +08:00
apoltavtsev
248c7c6d74 feat(TESB-28826) Secure Password - Publish job with Password 2020-07-14 19:50:59 +02:00
wang wei
a2a3eb2fbf fix(TDI-44548): tacokit components can't be loaded (#4931) 2020-07-14 16:46:30 +08:00
msheverda-talend
0592a4ddcc TESB-29664 Random port will be used when REST Engpoint in tRESTRequest is empty (#4903)
Co-authored-by: Maksym Sheverda <maksym.sheverda@synapse.com>
2020-07-14 10:50:14 +03:00
wang wei
bf30eba808 fix(TDI-44167): SCD ELT components to support Databricks Delta (#4919) 2020-07-13 16:38:24 +08:00
vyu-talend
e16c6f379e fix(TDI-44377):fix tfileinputdelimited utf8bom issue (#4853)
* fix(TDI-44377):fix tfileinputdelimited utf8bom issue

* fix(TDI-44377):add a checkbox

* fix(TDI-44377):skip bom to fix tfileinputdelimited

* fix(TDI-44300):skip bom to fix issue
2020-07-13 14:17:03 +08:00
vyu-talend
863a874fb2 fix(TDI-44300):skip bom to fix issue (#4897) 2020-07-13 14:14:32 +08:00
wang wei
78b2702600 fix(TDI-44506): Failed to build job on TOS (#4900) 2020-07-13 10:16:59 +08:00
Roman
18a18ce336 feat(TDI-38246): add new auth type to Google Storage family (#4839)
* feat(TDI-38246): add new auth type to tGSConnection

* feat(TDI-38246): add auth to UI | add dependencies

* feat(TDI-38246): add new auth type to tGSBucketCreate

* feat(TDI-38246): add new auth type to tGSBucketDelete | small fixes

* feat(TDI-38246): add new auth type to tGSBucketExist

* feat(TDI-38246): add new auth type to tGSBucketList

* feat(TDI-38246): remove projectId from tGSBucketList

* feat(TDI-38246): add new auth type to tGSClose

* feat(TDI-38246): add filemap generation for blob

* feat(TDI-38246): add new auth type to tGSPut

* feat(TDI-38246): add new auth type to tGSDelete

* feat(TDI-38246): add new auth type to tGSList

* feat(TDI-38246): add new auth type to tGSGet

* feat(TDI-38246): add new auth type to tGSCopy

* feat(TDI-38246): remove projectID for service account auth type

* feat(TDI-38246): rename component properties

* feat(TDI-38246): extend locations for bucket creation

* feat(TDI-38246): remove dual locations

* feat(TDI-38246): change default auth type

* feat(TDI-38246): add migration task

* feat(TDI-38246): remove unused var

* feat(TDI-38246): improve migration task

* feat(TDI-38246): clean code

* feat(TDI-38246): correct showing some params when use exist connection

* feat(TDI-38246): mark all HMAC keys as deprecated

* feat(TDI-38246): avoid blob directories
2020-07-10 18:32:09 +03:00
kjwang
51d66a086e Fix: TUP-27545 Update context var name for jdbc drivers can not be updated to job automatically. (#4883)
* Fix: TUP-27545 Update context var name for jdbc drivers can not be
updated to job automatically.
https://jira.talendforge.org/browse/TUP-27545
2020-07-10 14:13:09 +08:00
Roman
93735786d2 feat(TDI-37016): Add dynamic support for tVerticaBulkExec (#4484) (#4884) 2020-07-09 15:32:05 +03:00
clesaec
006b9c00b9 fix(TDI-44455) : log dependencies for studio-integration module (#4874) 2020-07-09 11:28:25 +02:00
jiezhang-tlnd
4790a880e1 feat(TUP-27680)Can't use the database "MAPPING" parameter to decide the component UI if show (#4879)
* feat(TUP-27680)Can't use the database "MAPPING" parameter to decide the
component UI if show
https://jira.talendforge.org/browse/TUP-27680

* feat(TUP-27680)Can't use the database "MAPPING" parameter to decide the
component UI if show

* feat(TUP-27680)add junits
2020-07-09 17:14:06 +08:00
jiezhang-tlnd
a96fdec0a7 add junit caused by TDI-44284 (#4906) 2020-07-09 14:49:09 +08:00
hcyi
bde9640722 fix(TUP-27352):Traces enabled by default when using Memory Run (#4770) 2020-07-08 09:46:28 +08:00
Dmytro Grygorenko
a93372f60e fix(TDI-44355): Add contributor's real name. (#4891) 2020-07-07 13:13:16 +03:00
pyzhou
d1559451e4 feat(TDI-44284):Rename old tEXAxxx connectors in tExasolxxx (#4842)
* feat(TDI-44284):Rename old tEXAxxx connectors in tExasolxxx

* reformat and change filter

* reformat

* migration task

* change name

* change filter to tExasolConnection

* fix db unify

* fix(TDI-44284):rename EXA

* debug

* debug2

* debug3
2020-07-07 16:05:16 +08:00
vdrokov
6abc8d737d fix(TESB-29489): Compile error occurred in code if the route version is different with the routelet version (#4830) 2020-07-03 09:56:31 +03:00
hzhao-talendbj
69c9182794 Hengzhao/backport73/tup 27070 (#4786)
* Tup-27070

* TUP-27070  fix cross join cases (#4772)
2020-07-03 09:43:02 +08:00
jiezhang-tlnd
dc62c40afe fix(TUP-27588)tCreateTable data types mapping in the schema editor is (#4820) (#4864)
always Mysql after dragging and dropping from database Metadata
https://jira.talendforge.org/browse/TUP-27588
2020-07-02 17:28:13 +08:00
pyzhou
f7952374e4 Pyzhou/tdi 43899 jdbcscdelt type0 (#4835)
* feat(TDI-43899):tJDBCSCDELT type0

* mysql insert type0

* fix log

* snowflake type 0

* mssql type0

* exasol type 0

* postgres type0

* fix bug for exasol

* oracle type 0

* correct typo

* fix bug of oracle

* type correction
2020-07-02 16:46:43 +08:00
Jane Ding
de3f85fc35 Hwang/tup 26059 (#4436) (#4876)
* bugfix(TUP-26059):tELTPostgresqlMap lost its "DB Type" column for which
the value should be "BPCHAR"

Co-authored-by: hwang-talend <hwang@talend.com>
2020-07-02 14:48:09 +08:00
chmyga
e62819fd76 fix(TDI-44384): set csv options (#4821)
Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-07-02 08:57:04 +03:00
Dmytro Grygorenko
adca838561 fix(TDI-44412): Implemented optional component fail when no files found by mask. (#4858) 2020-06-30 14:16:09 +03:00
clesaec
e87cfa18a1 Clesaec/tcomp 1733 log4j pb (#4867)
* fix(TCOMP-1733) log dep pb
2020-06-29 18:03:22 +02:00
clesaec
961e39ef3b Clesaec/tdi 44320 active if form 7.3 (#4851)
* fix(TDI-44320) : change page creation method
2020-06-26 16:27:37 +02:00
OleksiiNimych
205d575bab feat(TDI-44039): internal_ip_only for tGoogleDataprocManage (#4763) 2020-06-25 12:04:53 +03:00
mbasiuk-talend
15a25708b6 Mbasiuk/tdi 44165 add null marker field (#4768)
* fix(TDI-44165): add checkbox and text property

* fix(TDI-44165): fix indentation

* fix(TDI-44165): fix indentation v2

* fix(TDI-44165): update nullMarke

* fix(TDI-44165): add null marker customization for BigqueryOB
2020-06-24 19:35:26 +03:00
Richard Lecomte
4468d89b0e TDI-44342 : Ignore microseconds in BigQuery datetime (#4798)
* TDI-44342 : Ignore microseconds in BigQuery datetime

* TDI-44342 : Ignore microseconds in BigQuery datetime

Fixed for BigQuerySQLRow component
2020-06-24 14:13:28 +02:00
OleksiiNimych
3a25678438 fix(TDI-44378): add correct generation of create table sql for Sybase (#4815) 2020-06-24 12:58:18 +03:00
Jane Ding
115ddb1b05 fix(TUP-27517):Implicit Context is not working with the Field Separator (#4832)
* fix(TUP-27517):Implicit Context is not working with the Field Separator
as ~~#*#~~
https://jira.talendforge.org/browse/TUP-27517

* fix(TUP-27517):Implicit Context is not working with the Field Separator
as ~~#*#~~
https://jira.talendforge.org/browse/TUP-27517

* fix(TUP-27517):Implicit Context is not working with the Field Separator
as ~~#*#~~
https://jira.talendforge.org/browse/TUP-27517
2020-06-24 14:30:20 +08:00
Chao MENG
e6a755129f fix(TUP-27497): [7.3.1] tPreJob with tHDFSConnection with kerberized (#4850)
fix(TUP-27497): [7.3.1] tPreJob with tHDFSConnection with kerberized cluster fails with simple authentication error
https://jira.talendforge.org/browse/TUP-27497
2020-06-23 18:28:10 +08:00
Chao MENG
1938ecc8af fix(TUP-27497): [7.3.1] tPreJob with tHDFSConnection with kerberized cluster fails with simple authentication error (#4805)
* fix(TUP-27497): [7.3.1] tPreJob with tHDFSConnection with kerberized
cluster fails with simple authentication error
https://jira.talendforge.org/browse/TUP-27497

* fix(TUP-27497): [7.3.1] tPreJob with tHDFSConnection with kerberized
cluster fails with simple authentication error
https://jira.talendforge.org/browse/TUP-27497
2020-06-23 09:33:29 +08:00
Dmytro Sylaiev
b15d901b27 Dsylaiev/tdi 44109 reintegrate feature for 7.3 (#4834)
* feat(TDI-44109): Regain use ftps library for FTP mode (#4737)

* feat(TDI-44109): Use ftps library for FTP mode

* fix(TDI-44109): fix compile error

* fix(TDI-44109): Fix compile error with the tFTPFileList

* feat(TDI-44109): Use ftps library for FTP mode

* Fix compile error for ftp protocol

* Fix log information for ftp protocol

* Fix compile error for sftp in tFTPDelete component

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>

* fix(TDI-44109): Apply ZhaoJin's fix for FTPGet and Put

* fix(TDI-44109): Fix "move dir" pb on tFTPDelete

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-06-22 11:51:31 +03:00
Dmytro Grygorenko
7f92ad83a1 fix(TDI-44355): Backport to 'maintenance/7.3' branch (#4844) 2020-06-22 11:23:07 +03:00
Jean-Philippe HAUTIN
66b0cf243b fix(TBD-10799): HDI4.0: DI tHiveConnection does not show any fields (#4838) 2020-06-18 12:17:16 +02:00
AlixMetivier
e8ec4d0337 fix(TBD-10454): fix hadoop conf in context with TMC (#4833) 2020-06-18 08:52:06 +02:00
Dmytro Sylaiev
1e30a7ea90 feat(TDI-44109): Revert Use ftps library for FTP mode (#4737)
This reverts commit c484ce75b4.
2020-06-17 10:47:18 +03:00
wang wei
debbd1107f fix(TDI-44380): Imported JDBC job has compile error : extra quotes around table name(#4810) 2020-06-16 13:18:14 +08:00
Jane Ding
35d279a32a fix(TUP-27509):[bug] testcase listed everytime when click "Detect and… (#4797)
* fix(TUP-27509):[bug] testcase listed everytime when click "Detect and (#4796)

update all jobs"
https://jira.talendforge.org/browse/TUP-27509

* Fix: TUP-27513 [Bug] Studio update job context variable name, the
testcase will add another context variable as built-in mode
https://jira.talendforge.org/browse/TUP-27513

Co-authored-by: kjwang-talend <kjwang@talend.com>
2020-06-12 15:58:05 +08:00
Roman
6416c9c9bb Revert "feat(TDI-37016): Add dynamic support for tVerticaBulkExec (#4484)" (#4803)
This reverts commit cc6a1bc1da.
2020-06-11 15:17:31 +03:00
Jean-Philippe HAUTIN
e03637981c fix(TBD-10661): HDI4.0: set output format to tsv in beeline (#4790) 2020-06-10 21:57:26 +02:00
chmyga
52c5ef7dfa Dchmyga/tdi 44281 fix missing log4j lib t set proxy (#4794)
* fix(TDI-44281): NoClassDefFound for tSetProxy

* Use slf4j instead of log4j in talend-proxy lib

* fix(TDI-44281): NoClassDefFound for tSetProxy

* Update talend-proxy lib version in components

* fix(TDI-44281): NoClassDefFound for tSetProxy

* Update talend-proxy lib for download during build process

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-06-10 15:20:16 +03:00
Laurent BOURGEOIS
31b3974f81 fix(TBD-10285):Add contains predicate to expressions (#4613) (#4795) 2020-06-10 12:50:26 +02:00
wang wei
0507e76d5e fix(TDI-44172): Observability: Create new "component execution" event and emit it frequently (#4765) 2020-06-10 16:36:30 +08:00
Dmytro Sylaiev
c484ce75b4 feat(TDI-44109): Use ftps library for FTP mode (#4737)
* feat(TDI-44109): Use ftps library for FTP mode

* fix(TDI-44109): fix compile error

* fix(TDI-44109): Fix compile error with the tFTPFileList

* feat(TDI-44109): Use ftps library for FTP mode

* Fix compile error for ftp protocol

* Fix log information for ftp protocol

* Fix compile error for sftp in tFTPDelete component

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-06-10 09:56:07 +03:00
Zhiwei Xue
212c74ebb4 fix(TUP-27315):Patch installation issues for zero install CI (#4782)
* fix(TUP-27315):Patch installation issues for zero install CI

* fix(TUP-27315): re-deploy tacokit jars before server start.

* Revert "fix(TUP-27315):Patch installation issues for zero install CI"

This reverts commit 0005265955.

* fix(TUP-27315): log errors when copy car dependencies failed rather than
block the process

* fix(TUP-27315): log errors when copy car dependencies failed rather than
block the process
2020-06-10 09:31:38 +08:00
Roman
489125e931 fix(TDI-44128): Update axis2 dependency CVE (#4684) 2020-06-09 12:46:58 +03:00
Sizhao Liu
c0b2bae68e feat(TUP-27160): searchable connectors (for 7.3) (#4778)
* feat(TUP-27157): searchable stitch connectors

* fix the item selection issue in palette + load latest UTM params + i18n

* add category for stitch connectors + exclude overlapping connectors

* add timeout for resource loading
2020-06-08 10:59:03 +08:00
pyzhou
1f6dd38d99 fix(TDI-44196):Jdbc use origin name for dynamic (#4779) 2020-06-05 17:47:51 +08:00
chmyga
954a123f66 fix(TDI-44247): tBigQueryInput illegal operation (#4722)
* Get location from jobreference for OAuth 2.0 authentication mode

Co-authored-by: Dmytro Chmyga <dmytro.chmyga@synapse.com>
2020-06-05 11:22:17 +03:00
vyu-talend
a40d2a2379 Vyu/tdi 44214 change default driver in mssql (#4724)
* feat(TDI-44214):Change default driver to Microsoft official

* feat(tdi-44214):add migration task.

* feat(TDI-44214):modify the plugin.xml file.

* feat(TDI-44214):add tdq components in migration task

* feat(TDI-44214):change migration task.
2020-06-05 10:58:22 +08:00
jiezhang-tlnd
e20d0e57b5 feat(TUP-27259)Change default driver to Microsoft official for MSSql in Metadata (#4736)
* feat(TUP-27259)Change default driver to Microsoft official for MSSql in
Metadata
https://jira.talendforge.org/browse/TUP-27259

* Update plugin.xml

* feat(TUP-27259)Change default driver to Microsoft official for MSSql in
Metadata
https://jira.talendforge.org/browse/TUP-27259

* feat(TUP-27259)Change default driver to Microsoft official for MSSql in
Metadata
2020-06-05 09:22:09 +08:00
pyzhou
e14e19e306 feat(TDI-44216):Redshift partner tracker (#4747)
* feat(TDI-44216):Redshift partner tracker

* add tRedshiftUnload

fix bug

* add tracker for tCreateTable
2020-06-04 17:16:57 +08:00
bhe-talendbj
94534daf40 fix(TUP-27053): backport (#4766) 2020-06-04 11:21:04 +08:00
OleksiiNimych
16e8c9a4fd fix(TDI-44115): remove xmlrpc_3.1.2 from studio (#4753) 2020-06-03 17:55:12 +03:00
OleksiiNimych
59f80736a2 fix(TDI-44115): update xmlrpc-client version (#4711) 2020-06-03 17:54:21 +03:00
OleksiiNimych
511c3ccbd0 fix(TDI-44104): update xmlrpc-common version (#4703) 2020-06-03 17:53:34 +03:00
undx
dbf67ed4bb fix(TDI-44174): Component manager is not thread safe
- this fixes the case when run as independent job is checked and the
  m2.repository automatically defaults to user's m2 in exported job.
- issue on singleton will be fix by component-runtime upgrade.
2020-06-03 09:57:11 +02:00
Emmanuel GALLOIS
29a09a75bc fix(TDI-44174): Component manager is not thread safe (#4714)
* fix(TDI-44174): Component manager is not thread safe

- this fixes the case when run as independent job is checked and the
  m2.repository automatically defaults to user's m2 in exported job.
- issue on singleton will be fix by component-runtime upgrade.

(cherry picked from commit 09386aa841)
2020-06-03 09:53:56 +02:00
Dmytro Grygorenko
a73c156a6a fix(TDI-30505): Backport fix from "master" (#4761) 2020-06-01 16:02:30 +03:00
bhe-talendbj
790fe2ccb4 fix(TUP-27053): fix junit test case (#4676) (#4762) 2020-06-01 17:42:12 +08:00
hcyi
b507ddda61 fix(TUP-26683): Inputs label in documentation not the same as (#4641) (#4749)
FlowDiagram
2020-06-01 11:23:08 +08:00
OleksiiNimych
6512df89ad fix(TDI-44169): update xmlrpc-client version (#4710) 2020-05-29 14:53:22 +03:00
OleksiiNimych
a5043f56e8 fix(TDI-44105): update xmlrpc-common version (#4699) 2020-05-29 14:52:49 +03:00
Dmytro Grygorenko
c8ef5195f8 fix(TDI-44269): Set Outline ERROR_MESSAGE values. (#4743) 2020-05-29 14:02:27 +03:00
Chao MENG
2c2bb17695 feat(TUP-26569): merge known conflicts automatically (#4757)
https://jira.talendforge.org/browse/TUP-26569
2020-05-29 10:59:30 +08:00
kjwang
95400d2a90 Kjwang/feat tup 26288 enhance context b73 (#4746)
Fix TUP-26288 Enhance context propagation over reference project
https://jira.talendforge.org/browse/TUP-26288
2020-05-27 17:11:10 +08:00
wang wei
39cf036547 fix(TDI-43833): tDBBulkExec bug when using PostGress driver (#4701) 2020-05-27 14:21:56 +08:00
Zhiwei Xue
2a39dcaab8 fix(TUP-27262): fix regression from tuj of wrong param location. 2020-05-27 12:43:25 +08:00
Zhiwei Xue
34e490dd0f fix(TUP-27199): revert previous fix and remove wrapping space for
condition.
2020-05-26 11:09:58 +08:00
hwang-talend
d1f7dea73c Hwang/tup 27199 (#4692) (#4695)
Hwang/tup 27199  bugfix(TUP-27199):DB version of sybase isn't hide for other database
on
tCreateTable
2020-05-25 10:40:05 +08:00
Zhiwei Xue
c767870d82 fix(TUP-27262): tAzureAdlsGen2Output component fails with class not
found error
2020-05-22 21:42:38 +08:00
Roman
cc6a1bc1da feat(TDI-37016): Add dynamic support for tVerticaBulkExec (#4484) 2020-05-21 10:20:22 +03:00
wang wei
fee90cdd2f fix(TDI-44066): Illegal argument exception in tsapbapi name field (#4668) 2020-05-21 11:12:46 +08:00
Roman
b8e0a55fb7 fix(TDI-43812): filemask not supported by tFTPRename (#4680) 2020-05-20 15:57:07 +03:00
hzhao-talendbj
447e26d05c Hengzhao/backport73/tup 26476 (#4709)
* hengzhao backport73 tup-36467

* Tup-26476 fix missing filterKind issue (#4609)
2020-05-20 20:04:45 +08:00
Mike Yan
8a24135a74 feat(TESB-24998): Interpret context variable at endpoint for Data Services and Routes (#4708)
* feat(TESB-24998): Add context bean for tRESTReuest

* fix(TESB-24998): Fix runtime running error with quotes in url header
2020-05-20 17:41:15 +08:00
hcyi
316f2f7e96 fix(TUP-26924):tELTXXX: link name won't be updated after updating (#4697)
"default table name" in tELTInput.
2020-05-20 14:32:16 +08:00
Christophe Le Saec
9693179a6e Merge branch 'maintenance/7.3' of https://github.com/Talend/tdi-studio-se into maintenance/7.3 2020-05-19 11:18:41 +02:00
Christophe Le Saec
567abbf3d4 fix(TDI-44191) : change module for tfilecopy 2020-05-19 11:17:05 +02:00
SunChaoqun
9a3617679d TESB-27806:Expose Talend Microservices Metrics in a monitoring system like Prometheus(#4700) 2020-05-19 10:54:42 +08:00
jzhao
215cf5fe8d feat(TDI-44179):support copy command in Azure synapse bulk load (#4681)
* feat(TDI-44179):support copy command in Azure synapse bulk load



* fix(TDI-44179):create table options



* fix(TDI-44179):add column to source field mapping



* fix(TDI-44179):change some parameter to list



* fix(TDI-44179):change default load method and add migrationtask



* fix(TDI-44179):only show "error file location" for csv file format



* fix(TDI-44179):fix some issue



* hide error file location for shared access key

* support context for field index

* correct sas token field
2020-05-18 18:45:06 +08:00
AlixMetivier
21be67d243 fix(TBD-10454): with TMC strip timestamp to get hadoop conf (#4679) (#4689) 2020-05-14 15:36:20 +02:00
Zhiwei Xue
717623b206 fix(TUP-26961):When save/delete a test case, it might break the poms/CI (#4656) 2020-05-14 10:16:24 +08:00
hcyi
ea13d1115d fix(TUP-27000):Talend Salesforce Einstein connector Repository (#4671)
connection issue in Talend 7.3
2020-05-14 10:06:25 +08:00
jzhao
af52c4df65 feat(TDI-43719):Enhance Azure SQL data warehouse (#4492)
* feat(TDI-43719):rename tSQLDWHXXX component to tAzureSynapseXXX

* feat(TDI-43719):rename the reference

* feat(TDI-43719):support adls gen2 in tAzureSynapseBulkExec

* feat(TDI-43719):add migration task

* feat(TDI-43719):change the family name

* change the family name and i18n which include DWH
2020-05-13 18:22:47 +08:00
wang wei
e934d2112c fix(TDI-44159): Data viewer on tSybaseInput get error in context model with sybase 16 anywhere database(#4661) 2020-05-13 14:42:53 +08:00
Richard Lecomte
385843d3be TDI-43935 : MsSQLRow NString type for prepared statement (#4651) 2020-05-12 13:11:01 +02:00
Dmytro Sylaiev
aae6ad44b5 Dsylaiev/tdi 39575 active directory for mssql (#4584)
* feat(TDI-39575): Add AD auth support for AzureSynapse

* feat(TDI-39575): Add AD Auth support for MsSQL components

* feat(TDI-39575): Provide AD feature for tCreateTable
2020-05-12 13:21:50 +03:00
undx
a5e1f73458 fix(TCOMP-1681): fix Windows path case 2020-05-12 10:31:40 +02:00
vyu-talend
47cebcbddf Vyu/tdi 43756 fix bug in sqsclient (#4648)
* feat(TDI-43756):extends STS AssumeRole fields.

* feat(TDI-43756):extends AssumeRole fields(v2).

* feat(TDI-43756):set Session duration not required.

* feat(TDI-43756):

* feat(TDI-43756):extends fields for SQS

* feat(TDI-43756):fix the mistakes.

* feat(TDI-43756):extends fields for tAmazonEMRXXX,tAamzonRedshiftManage.

* feat(TDI-43756):optimize code.

* feat(TDI-43756):retrieve changes on S3connection.xml

* feat(TDI-43756):fix duplicated variables.

* feat(TDI-43756):fix same error

* feat(TDI-43756):optimize codes.

* feat(TDI-43756):use default region(US_WEST_2) for sts.

* feat(TDI-43756):change default to US-EAST-1.

* feat(TDI-43756):fix bug in SQSClient.

* feat(TDI-43756):fix bug in CredentialsProvider.
2020-05-12 16:09:38 +08:00
Jane Ding
0efcfa2983 bugfix(TUP-26958):Test case: possible to remove input / output node (#4633) (#4674)
bugfix(TUP-26958):Test case: possible to remove input / output node

Co-authored-by: hwang-talend <hwang@talend.com>
2020-05-12 09:25:57 +08:00
vyu-talend
76b0a7a979 Fix(TDI-43770):upgrade httpcore,httpclient,commons-logging,codec
Conflicts:
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftConnection/tRedshiftConnection_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftInput/tRedshiftInput_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftOutput/tRedshiftOutput_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftRow/tRedshiftRow_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftUnload/tRedshiftUnload_java.xml
2020-05-11 17:11:58 +08:00
vyu-talend
045d5df5df feat(TDI-43754): upgrade AWS SDK. (#4520)
Conflicts:
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftConnection/tRedshiftConnection_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftInput/tRedshiftInput_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftOutput/tRedshiftOutput_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftRow/tRedshiftRow_java.xml
	main/plugins/org.talend.designer.components.localprovider/components/tRedshiftUnload/tRedshiftUnload_java.xml
2020-05-11 17:11:49 +08:00
vyu-talend
453ca6e25a feat(TDI-43756):extends STS AssumeRole fields for S3. (#4547)
* feat(TDI-43756):extends STS AssumeRole fields.

* feat(TDI-43756):extends AssumeRole fields(v2).

* feat(TDI-43756):set Session duration not required.

* feat(TDI-43756):

* feat(TDI-43756):extends fields for SQS

* feat(TDI-43756):fix the mistakes.

* feat(TDI-43756):extends fields for tAmazonEMRXXX,tAamzonRedshiftManage.

* feat(TDI-43756):optimize code.

* feat(TDI-43756):retrieve changes on S3connection.xml

* feat(TDI-43756):fix duplicated variables.

* feat(TDI-43756):fix same error

* feat(TDI-43756):optimize codes.

* feat(TDI-43756):use default region(US_WEST_2) for sts.

* feat(TDI-43756):change default to US-EAST-1.
2020-05-11 17:11:40 +08:00
Zhiwei Xue
844d9e6c29 chore(TUP-26710):update version (#4672)
* chore(TUP-26710):update version

* Set version to 7.3.1 PATCH

* chore(TUP-26710):update version for feature.xml

Co-authored-by: Jenkins Continuous Build server <jenkins@tal-rd02.talend.lan>
2020-05-11 17:03:29 +08:00
hzhao-talendbj
9d218fbecf bugfix(TUP-26186):Use of the operators =, <=, <, >=, >, <> instead of (#4639) (#4664)
* bugfix(TUP-26186):Use of the operators =, <=, <, >=, >, <> instead of
EQ, LE, LT, GE, GT, NE

* bugfix(TUP-26186):Use of the operators =, <=, <, >=, >, <> instead of
EQ, LE, LT, GE, GT, NE

* bugfix(TUP-26186):Use of the operators =, <=, <, >=, >, <> instead of
EQ, LE, LT, GE, GT, NE

Co-authored-by: hwang-talend <hwang@talend.com>
2020-05-09 09:56:00 +08:00
Jane Ding
2e2c5f810d feat(TUP-16546)Prompt users for new password in Studio Connection when (#4508) (#4657)
* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD or LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546)Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546):Prompt users for new password in Studio Connection when
AD or LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546

* feat(TUP-16546):Prompt users for new password in Studio Connection when
AD/LDAP credentials change in TAC
https://jira.talendforge.org/browse/TUP-16546
2020-05-06 00:10:34 +08:00
Jane Ding
9c25ce552b fix(TUP-26639):Talend Cloud studio Issue -Changes to Context Parameter (#4653) (#4658)
Names is not reflected on the Connection for Snowflake
https://jira.talendforge.org/browse/TUP-26639
2020-05-04 19:14:19 +08:00
pyzhou
a42b25c9bb fix(TDI-44130):move log to finally (#4652) 2020-04-30 17:31:26 +08:00
Jane Ding
7d6e7e1c85 fix(TUP-26639):Talend Cloud studio Issues - Group context and (#4634) (#4646)
* fix(TUP-26639):Talend Cloud studio Issues - Group context and
connections
https://jira.talendforge.org/browse/TUP-26639

* fix(TUP-26639):Talend Cloud studio Issues - Group context and
connections
https://jira.talendforge.org/browse/TUP-26639
2020-04-27 23:21:04 +08:00
Zhiwei Xue
cfabaf818a fix(TUP-26809):Job zip should not contain jars from testcase only (#4635) 2020-04-27 18:05:09 +08:00
undx
e2f426e39f feat(TCOMP-1681): Make ESB Studio job export TaCoKit compliant
- OSGi assembly templates
- TaCoKit Studio part
- OSGi resources
2020-04-24 17:26:33 +02:00
pyzhou
ba1bf20653 fix(TDI-44051):add the real value to the db name (#4621)
* fix(TDI-44051):add the real value to the db name

* set dbName as name when db column is null

* workaround

* change back name
2020-04-24 11:24:14 +08:00
sbovsunovskyi
063db841c0 fix(TDI-44070): Unable to execute the job, receiving "Failed to gener… (#4625)
* fix(TDI-44070): Unable to execute the job, receiving "Failed to generate code." when using tAmazonRedshiftManage component.

* fix(TDI-44070): Unable to execute the job, receiving "Failed to generate code." when using tAmazonRedshiftManage component.

(cherry picked from commit 0fecec9635)
2020-04-21 16:36:37 +03:00
OleksiiNimych
a6640f95a9 fix(TDI-43995): Support listing any path and file (#4611)
* fix(TDI-43995): Support listing any path and file

* fix(TDI-43995): Updated Apache Commons Lang library and added diamond operator

* fix(TDI-43995): Remove Apache Commons Lang library
2020-04-16 10:49:20 +03:00
hzhao-talendbj
d01f8a7da9 TUP-26388 fix missing DB context value (#4614) (#4619)
* TUP-26388 fix missing DB context value

* fix missing db value issue

* remove useless isTcompv0
2020-04-15 09:51:20 +08:00
Zhiwei Xue
f8e9e01e31 fix(TUP-26728):Missing spark dependencies when using test cases (#4602) 2020-04-09 16:05:15 +08:00
hzhao-talendbj
94ff584488 TUP-26388 fix SQL script's syntax issue (#4603) (#4604)
* TUP-26388  fix SQL script's syntax issue

* TUP-26388  fix SQL script's syntax issue
2020-04-09 10:09:47 +08:00
OleksiiNimych
1db094edd3 fix(TDI-43795): Hide Single Insert Query action and mark it deprecated (#4588) 2020-04-07 22:48:48 +03:00
hzhao-talendbj
8b9fe33a86 fix(TUP-26388)Amazon Redshift Driver class not working with Implicit (#4585) (#4594)
* fix(TUP-26388)Amazon Redshift Driver class not working with Implicit
Context

* fix(TUP-26388)Amazon Redshift Driver class not working with Implicit
Context
2020-04-07 15:42:30 +08:00
Zhiwei Xue
4782321f64 fix(TUP-26482):[Studio] Studio is very slow to build the job(the (#4546)
performance issue)
2020-04-03 12:49:38 +08:00
Zhiwei Xue
12b61b8d5f fix(TUP-26233):[7.0.1] Sometimes , the ZIP file created by a commandline (#4569)
PublishJob is not correct, missing item files.
2020-03-31 18:27:42 +08:00
wang wei
8b26ef7e15 fix(TDI-43918): fix the risk of drop table if exception appear (#4572) 2020-03-30 17:48:04 +08:00
wang wei
8ac9327375 fix(TDI-43895): Teradata metadata not populating the fields properly in components(#4570) 2020-03-29 20:01:08 +08:00
hcyi
9f4810e59e fix(TUP-26580):DI job with Big Data components fail when using the (#4564)
custom hadoop conf file location feature.
2020-03-27 11:10:53 +08:00
jiezhang-tlnd
0f89caccfb fix(TUP-25963)Spark jobs - LOG4J_RUN_LEVEL not available in item (#4562)
https://jira.talendforge.org/browse/TUP-25963
2020-03-27 11:01:12 +08:00
hzhao-talendbj
c16c8f71f8 fix column names are same (#4551) (#4567) 2020-03-26 19:22:48 +08:00
clesaec
7ab0f2015a fix(TDI-43785) - big query repeated value managed (#4557) 2020-03-25 09:29:57 +01:00
Jane Ding
af43cd8973 fix(TUP-26576):Warning pops up cannot connect to the user libraries (#4553) (#4554)
artifact repository set in administrator
https://jira.talendforge.org/browse/TUP-26576
2020-03-25 09:21:16 +08:00
Dmytro Sylaiev
4169210118 Dsylaiev/tdi 43810 add ignore service attr checkbox for xml (#4539)
* fix(TDI-43810): Fix for xml component

* fix(TDI-43810): Do not ignore service attr when checkbox is hidden

* fix(TDI-43810): Use hashset instead of arraylist

* fix(TDI-43810): Minor refactoring
2020-03-24 12:32:47 +02:00
Jane Ding
3d185eeb95 fix(TUP-26231)Relative path not working in Talend v721 (#4530) (#4550)
* fix(TUP-26231)Relative path not working in Talend v721
https://jira.talendforge.org/browse/TUP-26231

* fix(TUP-26231)Relative path not working in Talend v721
https://jira.talendforge.org/browse/TUP-26231

* fix(TUP-26231)Relative path not working in Talend v721
https://jira.talendforge.org/browse/TUP-26231

* fix(TUP-26231):Relative path not working in Talend v721
https://jira.talendforge.org/browse/TUP-26231
2020-03-24 17:44:49 +08:00
Richard Lecomte
85a8f83baa TDI-43764 : MsSQL NVARCHAR (#4518)
* TDI-43764 : MsSQL NVARCHAR

Use setNString method for NVARCHAR and NCHAR columns only for MS driver

* TDI-43764 : MsSQL NVARCHAR

Use setNString method for NTEXT columns only for MS driver
2020-03-23 09:32:41 +01:00
Richard Lecomte
d3faad612e TDI-43764 : MsSQL NVARCHAR (#4491)
* TDI-43764 : MsSQL NVARCHAR

Use setNString method for NVARCHAR and NCHAR columns

* TDI-43764 : MsSQL NVARCHAR

Use setNString method for NVARCHAR and NCHAR columns
2020-03-23 09:32:32 +01:00
SunChaoqun
059c211971 TESB-28130:Duplicate dependencies in POM.xml for routes lead to compile issues(#4534) 2020-03-18 11:29:44 +08:00
Dmytro Sylaiev
ce028761e5 Dsylaiev/tdi 43605 fix using existing connection with rest tcompv0 (#4463)
* fix(TDi-43605): Fix adding tcompv0 components into tRest globalMap copy (#4367)

* fix(TDi-43605): Fix adding tcompv0 components into tRest globalMap copy

* fix(TDI-43605): Add referenced connection into globalMap copy for marketo

* fix(TDI-43605): Fix getting snowflake connection for plenty javajet components
2020-03-10 16:22:12 +02:00
wang wei
8c2a503063 fix(TDI-43619): Null Value Treated as <TALEND_NULL> in Subjob (#4488)
* fix(TDI-43619): Null Value Treated as <TALEND_NULL> in Subjob

* fix(TDI-43619): fix for all types to avoid the parse warn log or message
when null

* fix the compiler issue for password field
2020-03-10 16:19:18 +08:00
kjwang
ca7263b7a1 fix(TUP-26189): [7.2.1] snowflake metadata connection with proxy failure (#4498) (#4503)
https://jira.talendforge.org/browse/TUP-26189
2020-03-05 14:49:20 +08:00
Zhiwei Xue
04d41a8421 fix(TUP-26132):Cannot build a job by CI (maven) after updating multiple (#4442)
job versions
2020-03-05 14:37:40 +08:00
Zhiwei Xue
d35506dce1 fix(TUP-26213):Compilation issue after migration to v7.2 (#4478) 2020-03-05 09:54:09 +08:00
kjwang
5e05ca8947 Kjwang/fix tup 26264 t elt map generated query has extra symbols (#4489) (#4500)
* Fix TUP-26264 tELTMap generated query has extra symbols generated when
more input mapping in component
https://jira.talendforge.org/browse/TUP-26264

* Fix TUP-26264 tELTMap generated query has extra symbols generated when
more input mapping in component
https://jira.talendforge.org/browse/TUP-26264
2020-03-04 11:58:55 +08:00
vyu-talend
51f8af8aed Fix(TDI-43704):update mail jar. 2020-03-03 14:28:27 +08:00
Jenkins Continuous Build server
360f1a2612 Set version to 7.3.2 SNAPSHOT 2020-02-28 13:00:24 +01:00
Irene Wang
4f668f6a46 Update NOTICE.txt to 2020 2020-02-19 12:45:17 +01:00
jiezhang-tlnd
f75fed8fd8 Jzhang/bugfix/tup 26192 patch731 (#4462)
* fix(TUP-26192)supply choice of log4j1 / 2

* fix(TUP-26192)

* fix tos

* fix tos
2020-02-18 22:06:42 +08:00
hcyi
812ac5399d fix(TUP-25229):revert the fix of "Use this Component" (#4460) 2020-02-18 17:53:55 +08:00
AlixMetivier
d917b791d0 remove useless migration task (#4452) (#4453) 2020-02-18 10:28:58 +01:00
Mike Yan
9bc4abb02d fix(TESB-28247): consumer job is missing feature.xml by CI (#4451)
* fix(TESB-28247): consumer job is missing feature.xml by CI

* fix(TESB-28247): consumer job is missing feature.xml by CI
2020-02-18 14:28:12 +08:00
Mike Yan
3daedf72de fix(TESB-28240): Use final name for osgi bundle build pom (#4449) 2020-02-18 09:34:41 +08:00
Jenkins Continuous Build server
854ac0d1b1 Set version to 7.3.1 2020-02-14 13:38:41 +01:00
2015 changed files with 68631 additions and 33776 deletions

1
.gitignore vendored
View File

@@ -11,3 +11,4 @@ main/plugins/org.talend.designer.codegen/resources/sparkstreaming_header.javajet
main/plugins/org.talend.designer.codegen/resources/sparkstreaming_subprocess_footer.javajet
main/plugins/org.talend.designer.codegen/resources/sparkstreaming_subprocess_header.javajet
main/plugins/org.talend.designer.codegen/resources/routelet.skeleton
.DS_Store

View File

@@ -4,5 +4,6 @@ Huge thanks to these people who contributed in making Talend product better:
- [jlolling](https://github.com/jlolling) (Jan -and Miss- Lolling)
- [kirkalicious](https://github.com/kirkalicious) (Kirk Tarou)
- [mrizzato](https://github.com/mrizzato) (Massimiliano Rizzato)
- [overpackaging](https://github.com/overpackaging) (Steeve Dreux)
- [brazabr](https://github.com/brazabr) (Thiago Benvenuto)
- [phoflack](https://jira.talendforge.org/secure/ViewProfile.jspa?name=phoflack) (Philip Hoflack)

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.abstractmap.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.alfrescooutput.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.business.diagram.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.codegen.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.components.exchange.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.core.nl</artifactId>

View File

@@ -584,13 +584,11 @@ EParameterName.jdbcURL=JDBC URL
EParameterName.driverJar=Driver jar
EParameterName.className=Class name
EParameterName.mappingFile=Mapping file
SetupProcessDependenciesRoutinesAction.title=Setup routine dependencies
SetupProcessDependenciesRoutinesAction.title=Setup Codes Dependencies
SetupProcessDependenciesRoutinesDialog.systemRoutineLabel=System routines
SetupProcessDependenciesRoutinesDialog.userRoutineLabel=User routines
PerformancePreferencePage.addAllSystemRoutines=Add all system routines to job dependencies, when creating a new job
PerformancePreferencePage.addAllUserRoutines=Add all user routines to job dependencies, when creating a new job
ShowRoutineItemsDialog.systemTitle=Select Sytem Routines
ShowRoutineItemsDialog.title=Select Routines
AbstractMultiPageTalendEditor_pleaseWait=Saving Please Wait....
DocumentationPreferencePage.use_css_template=Use CSS file as a template when export to HTML
DocumentationPreferencePage.css_file=CSS File

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.dbmap.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.documentation.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.fileoutputxml.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.gefabstractmap.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.mapper.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.maven.job.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.rowgenerator.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.runprocess.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.scd.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.webservice.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.xmlmap.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.expressionbuilder.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.presentation.onboarding.resource.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.presentation.onboarding.resource.tos.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.repository.ftp.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.repository.json.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.repository.nl</artifactId>

View File

@@ -1,5 +1,5 @@
NavigatorContent.contexts=Contexts
NavigatorContent.routines=Routines
NavigatorContent.routines=Global Routines
NavigatorContent.sqlTemplates=SQL Templates
NavigatorContent.documentation=Documentation
NavigatorContent.activation=di.fake.for.activation

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.repository.view.di.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.sbi.engines.client.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.spagic.engines.client.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.sqlbuilder.nl</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.business.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.fragment.jars.di.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.help.tos.components.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.rcp.branding.tos.feature</artifactId>

View File

@@ -6,7 +6,7 @@
<license url="http://www.example.com/license">[Enter License Description here.]</license>
<requires>
<import feature="org.eclipse.test" version="0.0.0" match="greaterOrEqual"/>
<import plugin="org.junit" version="0.0.0" match="greaterOrEqual"/>
<import plugin="org.junit" version="4.13.2" match="greaterOrEqual"/>
<import plugin="org.talend.commons.runtime" version="0.0.0" match="greaterOrEqual"/>
<import plugin="org.talend.commons.ui" version="0.0.0" match="greaterOrEqual"/>
<import plugin="org.talend.core" version="0.0.0" match="greaterOrEqual"/>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.rcp.branding.tos.test.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.repository.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.sbi.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.spagic.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.tos.components.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.tos.core.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.tos.feature</artifactId>

View File

@@ -16,11 +16,9 @@
</requires>
<plugin id="org.talend.libraries.apache" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.axis2" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.batik" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.chemistry" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.common" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.cxf" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.google" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.http" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.lucene" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.apache.xml" download-size="0" install-size="0" version="0.0.0"/>
@@ -52,5 +50,4 @@
<plugin id="org.talend.libraries.slf4j" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.xml" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.zmq" download-size="0" install-size="0" version="0.0.0"/>
<plugin id="org.talend.libraries.zookeeper" download-size="0" install-size="0" version="0.0.0"/>
</feature>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.tos.libraries.feature</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.abstractmap</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.alfrescooutput</artifactId>

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.business.diagram</artifactId>

View File

@@ -50,8 +50,11 @@ List<IConnection> allSubProcessConnection = codeGenArgument.getAllMainSubTreeCon
String cid = node.getUniqueName();
List<? extends INode> jobCatcherNodes = process.getNodesOfType("tJobStructureCatcher");
boolean enableLogStash = jobCatcherNodes != null && !jobCatcherNodes.isEmpty();
boolean logstashCurrent = !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend") && enableLogStash;
boolean jobCatcherExists = jobCatcherNodes != null && !jobCatcherNodes.isEmpty();
INode jobCatcherNode = jobCatcherExists ? jobCatcherNodes.get(0) : null;
boolean enableLogStash = !Boolean.getBoolean("deactivate_extended_component_log") && jobCatcherExists;
boolean logstashCurrent = enableLogStash && !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend");
if((codePart.equals(ECodePart.END))&&(stat || logstashCurrent)){
boolean iterateInVFComp = (node.getVirtualLinkTo() != null && node.getVirtualLinkTo() == EConnectionType.ITERATE);
@@ -86,11 +89,18 @@ if((codePart.equals(ECodePart.END))&&(stat || logstashCurrent)){
for(IConnection con:connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
for (INode jobStructureCatcher : jobCatcherNodes) {
%>
if(runStat.updateStatAndLog(execStat,enableLogStash,resourceMap,iterateId,"<%=con.getUniqueName()%>",2,0,
<%=jobStructureCatcher.getUniqueName()%>,"<%=sourceNodeId%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=node.getComponent().getName()%>","<%="REJECT".equals(con.getConnectorName()) ? "reject" : "output"%>")) {
"<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>","<%="REJECT".equals(con.getConnectorName()) ? "reject" : "output"%>")) {
<%=jobStructureCatcher.getDesignSubjobStartNode().getUniqueName() %>Process(globalMap);
}
<%
@@ -113,11 +123,18 @@ if((codePart.equals(ECodePart.END))&&(stat || logstashCurrent)){
for(IConnection con:connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
for (INode jobStructureCatcher : jobCatcherNodes) {
%>
if(runStat.log(resourceMap,iterateId,"<%=con.getUniqueName()%>",2,0,
<%=jobStructureCatcher.getUniqueName()%>,"<%=sourceNodeId%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=node.getComponent().getName()%>","<%="REJECT".equals(con.getConnectorName()) ? "reject" : "output"%>")) {
"<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>","<%="REJECT".equals(con.getConnectorName()) ? "reject" : "output"%>")) {
<%=jobStructureCatcher.getDesignSubjobStartNode().getUniqueName() %>Process(globalMap);
}
<%
@@ -150,6 +167,7 @@ if((codePart.equals(ECodePart.END))&&(stat || logstashCurrent)){
}
}
}
List<IMetadataTable> metadatas = node.getMetadataList();
if ((!node.isSubProcessStart())&&(NodeUtil.isDataAutoPropagated(node))) {
if (inputColName!=null) {
@@ -175,6 +193,117 @@ if((codePart.equals(ECodePart.END))&&(stat || logstashCurrent)){
}
}
}
//log runtime lineage
boolean enable_runtime_lineage_log = NodeUtil.isJobUsingRuntimeLineage(process) && jobCatcherExists && !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend");
if(enable_runtime_lineage_log) {//}
List<? extends IConnection> outConns = node.getOutgoingConnections();
if(!outConns.isEmpty()) {
%>
if(tos_count_<%=node.getUniqueName() %> == 0) {
<%
//}
}
for (IConnection conn : outConns) {
if(!conn.getLineStyle().equals(EConnectionType.FLOW_MAIN) && !conn.getLineStyle().equals(EConnectionType.FLOW_MERGE) && !conn.getLineStyle().equals(EConnectionType.FLOW_REF)) {
continue;
}
IMetadataTable metadata = conn.getMetadataTable();
if (metadata==null) {
continue;
}
List<IMetadataColumn> columns = metadata.getListColumns();
if(columns == null || columns.isEmpty()) {
continue;
}
%>
class SchemaUtil_<%=conn.getUniqueName()%>_<%=conn.getMetadataTable().getTableName()%> {
public java.util.List<java.util.Map<String, String>> getSchema(final <%=NodeUtil.getPrivateConnClassName(conn) %>Struct <%=conn.getName()%>) {
java.util.List<java.util.Map<String, String>> schema = new java.util.ArrayList<>();
if(<%=conn.getName()%> == null) {
return schema;
}
java.util.Map<String, String> field = null;
<%
for(IMetadataColumn column : columns){
if("id_Dynamic".equals(column.getTalendType())) {
%>
routines.system.Dynamic dynamic = <%=conn.getName()%>.<%=column.getLabel()%>;
if(dynamic != null) {
for(routines.system.DynamicMetadata metadata : dynamic.metadatas) {
field = new java.util.HashMap<>();
field.put("name", metadata.getName());
field.put("origin_name", metadata.getDbName());
field.put("iskey", "" + metadata.isKey());
field.put("talend_type", metadata.getType());
field.put("type", metadata.getDbType());
field.put("nullable", "" + metadata.isNullable());
field.put("pattern", metadata.getFormat());
field.put("length", "" + metadata.getLength());
field.put("precision", "" + metadata.getPrecision());
schema.add(field);
}
}
<%
continue;
}
String pattern = column.getPattern();
if(pattern == null || pattern.isEmpty() || pattern.equals("\"\"")) {
pattern = "\"\"";
}
%>
field = new java.util.HashMap<>();
field.put("name", "<%=column.getLabel()%>");
field.put("origin_name", "<%=column.getOriginalDbColumnName()%>");
field.put("iskey", "<%=column.isKey()%>");
field.put("talend_type", "<%=column.getTalendType()%>");
field.put("type", "<%=column.getType()%>");
field.put("nullable", "<%=column.isNullable()%>");
field.put("pattern", <%=pattern%>);
field.put("length", "<%=column.getLength()%>");
field.put("precision", "<%=column.getPrecision()%>");
schema.add(field);
<%
}
%>
return schema;
}
}
java.util.List<java.util.Map<String, String>> schema_<%=conn.getUniqueName()%>_<%=conn.getMetadataTable().getTableName()%> = new SchemaUtil_<%=conn.getUniqueName()%>_<%=conn.getMetadataTable().getTableName()%>().getSchema(<%=conn.getName()%>);
<%
INode target = conn.getTarget();
String targetNodeId = target.getUniqueName();
String targetNodeComponent = target.getComponent().getName();
%>
<%=jobCatcherNode.getUniqueName()%>.addConnectionSchemaMessage("<%=node.getUniqueName()%>","<%=node.getComponent().getName()%>",
"<%=targetNodeId%>","<%=targetNodeComponent%>", "<%=conn.getUniqueName()%>" + iterateId, schema_<%=conn.getUniqueName()%>_<%=conn.getMetadataTable().getTableName()%>);
<%=jobCatcherNode.getDesignSubjobStartNode().getUniqueName() %>Process(globalMap);
<%
}
if(!outConns.isEmpty()) {
//{
%>
}
<%
}
//{
}
//======================================TDI-17183 end=====================================
boolean traceCodeGenerated = false;
for (IConnection conn : node.getOutgoingConnections()) {

View File

@@ -8,6 +8,9 @@
org.talend.core.model.metadata.IMetadataColumn
org.talend.core.model.process.EConnectionType
org.talend.core.model.process.ElementParameterParser
org.talend.core.model.process.EParameterFieldType
org.talend.designer.core.model.components.EParameterName
org.talend.designer.core.model.components.ElementParameter
org.talend.designer.codegen.config.CodeGeneratorArgument
org.talend.core.model.utils.NodeUtil
org.talend.core.model.process.IConnectionCategory
@@ -140,10 +143,17 @@
connSet.addAll(node.getIncomingConnections(EConnectionType.FLOW_MAIN));
connSet.addAll(node.getIncomingConnections(EConnectionType.FLOW_MERGE));
List<? extends INode> jobCatcherNodes = node.getProcess().getNodesOfType("tJobStructureCatcher");
boolean enableLogStash = jobCatcherNodes != null && !jobCatcherNodes.isEmpty();
String cid = node.getUniqueName();
boolean logstashCurrent = !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend") && enableLogStash;
List<? extends INode> jobCatcherNodes = node.getProcess().getNodesOfType("tJobStructureCatcher");
boolean jobCatcherExists = jobCatcherNodes != null && !jobCatcherNodes.isEmpty();
INode jobCatcherNode = jobCatcherExists ? jobCatcherNodes.get(0) : null;
boolean enableLogStash = !Boolean.getBoolean("deactivate_extended_component_log") && jobCatcherExists;
boolean logstashCurrent = enableLogStash && !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend");
//about performance monitor, no way to support more than one job catcher component, also that is not necessary
final String subprocessName4Catcher = logstashCurrent ? jobCatcherNodes.get(0).getDesignSubjobStartNode().getUniqueName() : null;
if ((codePart.equals(ECodePart.BEGIN))&&(stat || logstashCurrent)&&connSet.size()>0) {
if(containsTPartitioner) {
@@ -181,13 +191,35 @@
if(!node.getComponent().useMerge()) {
if(stat && logstashCurrent) {
%>
runStat.updateStatAndLog(execStat,enableLogStash,iterateId,1,1<%for(IConnection con : connSet){%>,"<%=con.getUniqueName()%>"<%}%>);
if(runStat.update(execStat,enableLogStash,iterateId,1,1
<%for(IConnection con : connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
%>
,"<%=con.getUniqueName()%>","<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>"
<%}%>
)) {
<%=subprocessName4Catcher %>Process(globalMap);
}
<%
} else {
if(stat) {
%>
if(execStat){
runStat.updateStatOnConnection(iterateId,1,1<%for(IConnection con : connSet){%>,"<%=con.getUniqueName()%>"<%}%>);
runStat.updateStatOnConnection(iterateId,1,1
<%for(IConnection con : connSet){
%>
,"<%=con.getUniqueName()%>"
<%}%>
);
}
<%
}
@@ -195,7 +227,24 @@
if(logstashCurrent) {
%>
if(enableLogStash) {
runStat.log(iterateId,1,1<%for(IConnection con : connSet){%>,"<%=con.getUniqueName()%>"<%}%>);
if(runStat.updateLog(iterateId,1,1
<%for(IConnection con : connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
%>
,"<%=con.getUniqueName()%>","<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>"
<%}%>
)) {
<%=subprocessName4Catcher %>Process(globalMap);
}
}
<%
}
@@ -205,18 +254,62 @@
if(connection.getUniqueName().equals((String)codeGenArgument.getIncomingName())){
if(stat && logstashCurrent) {
%>
runStat.updateStatAndLog(execStat,enableLogStash,iterateId,1,1<%for(IConnection con : connSet){if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>,"<%=con.getUniqueName()%>"<%}}%>);
if(runStat.update(execStat,enableLogStash,iterateId,1,1
<%for(IConnection con : connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>
,"<%=con.getUniqueName()%>","<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>"
<%}
}%>
)) {
<%=subprocessName4Catcher %>Process(globalMap);
}
<%
} else {
if(stat) {%>
if(execStat){
runStat.updateStatOnConnection(iterateId,1,1<%for(IConnection con : connSet){if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>,"<%=con.getUniqueName()%>"<%}}%>);
runStat.updateStatOnConnection(iterateId,1,1
<%for(IConnection con : connSet){
if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>
,"<%=con.getUniqueName()%>"
<%}
}%>
);
}
<%}
if(logstashCurrent) {%>
if(enableLogStash) {
runStat.log(iterateId,1,1<%for(IConnection con : connSet){if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>,"<%=con.getUniqueName()%>"<%}}%>);
if(runStat.updateLog(iterateId,1,1
<%for(IConnection con : connSet){
INode source = con.getSource();
String sourceNodeId = source.getUniqueName();
String sourceLabel = ElementParameterParser.getValue(source, "__LABEL__");
String sourceNodeLabel = ((sourceLabel==null || "__UNIQUE_NAME__".equals(sourceLabel) || sourceLabel.contains("\"")) ? sourceNodeId : sourceLabel.trim());
String targetLabel = ElementParameterParser.getValue(node, "__LABEL__");
String targetNodeLabel = ((targetLabel==null || "__UNIQUE_NAME__".equals(targetLabel) || targetLabel.contains("\"")) ? node.getUniqueName() : targetLabel.trim());
String sourceNodeComponent = source.getComponent().getName();
if(con.getUniqueName().equals((String)codeGenArgument.getIncomingName())){%>
,"<%=con.getUniqueName()%>","<%=sourceNodeId%>","<%=sourceNodeLabel%>","<%=sourceNodeComponent%>","<%=node.getUniqueName()%>","<%=targetNodeLabel%>","<%=node.getComponent().getName()%>"
<%}
}%>
)) {
<%=subprocessName4Catcher %>Process(globalMap);
}
}
<%}
}
@@ -245,12 +338,107 @@
<%
log.startWork();
log.logCompSetting();
boolean enable_runtime_lineage_log = NodeUtil.isJobUsingRuntimeLineage(node.getProcess()) && jobCatcherExists && !cid.startsWith("tJobStructureCatcher") && !cid.startsWith("talend");
if(enable_runtime_lineage_log) {
%>
class ParameterUtil_<%=cid%>{
public java.util.Map<String, String> getParameter() throws Exception{
java.util.Map<String, String> component_parameters = new java.util.HashMap<>();
<%
java.util.Set<org.talend.core.model.process.EParameterFieldType> ignoredParamsTypes = new java.util.HashSet<org.talend.core.model.process.EParameterFieldType>();
ignoredParamsTypes.addAll(
java.util.Arrays.asList(
org.talend.core.model.process.EParameterFieldType.SCHEMA_TYPE,
org.talend.core.model.process.EParameterFieldType.SCHEMA_REFERENCE,
org.talend.core.model.process.EParameterFieldType.LABEL,
org.talend.core.model.process.EParameterFieldType.EXTERNAL,
org.talend.core.model.process.EParameterFieldType.MAPPING_TYPE,
org.talend.core.model.process.EParameterFieldType.IMAGE,
org.talend.core.model.process.EParameterFieldType.TNS_EDITOR,
org.talend.core.model.process.EParameterFieldType.WSDL2JAVA,
org.talend.core.model.process.EParameterFieldType.GENERATEGRAMMARCONTROLLER,
org.talend.core.model.process.EParameterFieldType.GENERATE_SURVIVORSHIP_RULES_CONTROLLER,
org.talend.core.model.process.EParameterFieldType.REFRESH_REPORTS,
org.talend.core.model.process.EParameterFieldType.BROWSE_REPORTS,
org.talend.core.model.process.EParameterFieldType.PALO_DIM_SELECTION,
org.talend.core.model.process.EParameterFieldType.GUESS_SCHEMA,
org.talend.core.model.process.EParameterFieldType.MATCH_RULE_IMEX_CONTROLLER,
org.talend.core.model.process.EParameterFieldType.MEMO_PERL,
org.talend.core.model.process.EParameterFieldType.DBTYPE_LIST,
org.talend.core.model.process.EParameterFieldType.VERSION,
org.talend.core.model.process.EParameterFieldType.TECHNICAL,
org.talend.core.model.process.EParameterFieldType.ICON_SELECTION,
org.talend.core.model.process.EParameterFieldType.JAVA_COMMAND,
org.talend.core.model.process.EParameterFieldType.TREE_TABLE,
org.talend.core.model.process.EParameterFieldType.VALIDATION_RULE_TYPE,
org.talend.core.model.process.EParameterFieldType.DCSCHEMA,
org.talend.core.model.process.EParameterFieldType.SURVIVOR_RELATION,
org.talend.core.model.process.EParameterFieldType.REST_RESPONSE_SCHEMA_TYPE,
org.talend.core.model.process.EParameterFieldType.BUTTON
)
);
for(org.talend.core.model.process.IElementParameter ep : org.talend.core.model.utils.NodeUtil.getDisplayedParameters(node)){
if(!ep.isLog4JEnabled() || ignoredParamsTypes.contains(ep.getFieldType())){
continue;
}
ElementParameter p = (ElementParameter)ep;
Object pluginValue = p.getTaggedValue("org.talend.sdk.component.source");
if(pluginValue != null && String.class.cast(pluginValue).equalsIgnoreCase("tacokit")) {
try {
if (!(Boolean) org.talend.core.runtime.IAdditionalInfo.class.cast(p).func("isPersisted")) {
continue;
}
} catch (Exception ex) {
//do nothing
}
%>
<%@ include file="./tacokit_runtime_log.javajet"%>
<%
continue;
}
String name = ep.getName();
java.util.Set<String> ignoredParamsNames = new java.util.HashSet<String>();
ignoredParamsNames.add("SQLPATTERN_VALUE");
ignoredParamsNames.add("ADDITIONAL_INSERT_COLUMNS");
ignoredParamsNames.add("ADDITIONAL_UPDATE_COLUMNS");
ignoredParamsNames.add("SELECTION_TABLE");
ignoredParamsNames.add("DIFFER_MESSAGE");
ignoredParamsNames.add("NO_DIFFER_MESSAGE");
if(ignoredParamsNames.contains(name)) {
//do nothing
} else if(org.talend.core.model.process.EParameterFieldType.PASSWORD.equals(ep.getFieldType()) || org.talend.core.model.process.EParameterFieldType.HIDDEN_TEXT.equals(ep.getFieldType())){
//not log password
}else{
String value = org.talend.core.model.utils.NodeUtil.getRuntimeParameterValue(node, ep);
%>
component_parameters.put("<%=name%>", String.valueOf(<%=value%>));
<%
}
}
%>
return component_parameters;
}
}
<%=jobCatcherNode.getUniqueName()%>.addComponentParameterMessage("<%=node.getUniqueName()%>", "<%=node.getComponent().getName()%>",
new ParameterUtil_<%=cid%>().getParameter());
<%=jobCatcherNode.getDesignSubjobStartNode().getUniqueName() %>Process(globalMap);
<%
}
if(logstashCurrent) {
for (INode jobStructureCatcher : jobCatcherNodes) {
String label = ElementParameterParser.getValue(node, "__LABEL__");
String nodeLabel = ((label==null || "__UNIQUE_NAME__".equals(label) || label.contains("\"")) ? node.getUniqueName() : label.trim());
%>
if(enableLogStash) {
<%=jobStructureCatcher.getUniqueName() %>.addCM("<%=node.getUniqueName()%>", "<%=node.getComponent().getName()%>");
<%=jobStructureCatcher.getUniqueName() %>.addCM("<%=node.getUniqueName()%>", "<%=nodeLabel%>", "<%=node.getComponent().getName()%>");
<%=jobStructureCatcher.getDesignSubjobStartNode().getUniqueName() %>Process(globalMap);
}
<%

View File

@@ -24,6 +24,7 @@
org.talend.core.model.process.ProcessUtils
org.talend.core.model.components.IComponent
org.talend.core.model.components.EComponentType
org.talend.core.model.utils.NodeUtil
"
class="Footer"
skeleton="footer_java.skeleton"
@@ -108,10 +109,33 @@
exist_tSQLDB = true;
}
String[] tcompV0Components = new String[] {
"tAzureStorageConnection",
"tGoogleDriveConnection",
"tMarketoConnection",
"tSnowflakeConnection",
"tSalesforceConnection",
"tNetsuiteConnection",
"tMarkLogicConnection"
};
List<INode> tcompV0ComponentsList = new ArrayList<INode>();
for (String tcompV0Component : tcompV0Components) {
tcompV0ComponentsList.addAll((List<INode>)process.getNodesOfType(tcompV0Component));
}
boolean exist_tSCP = false;
List<INode> scpComponentsList = (List<INode>)process.getNodesOfType("tSCPConnection");
if (scpComponentsList.size() > 0) {
String parameterNames = "";
int scpsize = scpComponentsList.size();
if (scpsize > 0) {
exist_tSCP = true;
for (int i = 0; i < scpsize; i++) {
parameterNames += "\"conn_" + scpComponentsList.get(i).getUniqueName() + "\"";
if(i < scpsize-1){
parameterNames += ",";
}
}
}
boolean exist_tCassandra = false;
@@ -167,6 +191,8 @@
exist_tSalesforce = true;
}
List<INode> snowflakeComponentsList = (List<INode>)process.getNodesOfType("tSnowflakeConnection");
boolean exist_tSAP= false;
List<INode> sapComponentsList = (List<INode>)process.getNodesOfType("tSAPConnection");
if (sapComponentsList.size() > 0) {
@@ -258,7 +284,7 @@
}
%>
private PropertiesWithType context_param = new PropertiesWithType();
protected PropertiesWithType context_param = new PropertiesWithType();
public java.util.Map<String, Object> parentContextMap = new java.util.HashMap<String, Object>();
public String status= "";
@@ -447,7 +473,7 @@
lastStr = "";
}
}
enableLogStash = "true".equalsIgnoreCase(System.getProperty("monitoring"));
enableLogStash = "true".equalsIgnoreCase(System.getProperty("audit.enabled"));
<%if(isLog4jEnabled){%>
if(!"".equals(log4jLevel)){
@@ -501,8 +527,63 @@
break;
}
}
if(jobCatcherNode!=null) {
String location = ElementParameterParser.getValue(jobCatcherNode, "__LOCATION__");
boolean enableLogStash = !Boolean.getBoolean("deactivate_extended_component_log") && (jobCatcherNode!=null);
boolean enable_runtime_lineage_log = NodeUtil.isJobUsingRuntimeLineage(process) && (jobCatcherNode!=null);
if(enable_runtime_lineage_log) {
%>
java.util.Properties p_<%=jobCatcherNode.getUniqueName()%> = new java.util.Properties();
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("root.logger", "runtime");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("encoding", "UTF-8");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("application.name", "Talend Studio");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("service.name", "Talend Studio Job");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("instance.name", "Talend Studio Job Instance");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("propagate.appender.exceptions", "none");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("log.appender", "file");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("appender.file.path", "runtime.json");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("appender.file.maxsize", "52428800");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("appender.file.maxbackup", "20");
p_<%=jobCatcherNode.getUniqueName()%>.setProperty("host", "false");
final String runtime_dir_<%=jobCatcherNode.getUniqueName()%> = System.getProperty("runtime.lineage.outputpath");
final String runtime_path_<%=jobCatcherNode.getUniqueName()%> = System.getProperty("runtime.lineage.appender.file.path");
if(runtime_path_<%=jobCatcherNode.getUniqueName()%>==null || runtime_path_<%=jobCatcherNode.getUniqueName()%>.isEmpty()) {
if(runtime_dir_<%=jobCatcherNode.getUniqueName()%>!=null && !runtime_dir_<%=jobCatcherNode.getUniqueName()%>.isEmpty()) {
System.setProperty("runtime.lineage.appender.file.path",
new StringBuilder().append(runtime_dir_<%=jobCatcherNode.getUniqueName()%>)
.append((runtime_dir_<%=jobCatcherNode.getUniqueName()%>.endsWith("/") || runtime_dir_<%=jobCatcherNode.getUniqueName()%>.endsWith("\\")) ? "" : java.io.File.separator)
.append(projectName)
.append(java.io.File.separatorChar)
.append(jobName)
.append(java.io.File.separatorChar)
.append(jobVersion)
.append(java.io.File.separatorChar)
.append("runtime_log_")
.append(new java.text.SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date()))
.append(".json")
.toString()
);
}
}
System.getProperties().stringPropertyNames().stream()
.filter(it -> it.startsWith("runtime.lineage.") && !"runtime.lineage.outputpath".equals(it))
.forEach(key -> p_<%=jobCatcherNode.getUniqueName()%>.setProperty(key.substring("runtime.lineage.".length()), System.getProperty(key)));
<%if(isLog4j1Enabled) {%>
org.apache.log4j.Logger.getLogger(p_<%=jobCatcherNode.getUniqueName()%>.getProperty("root.logger")).setLevel(org.apache.log4j.Level.DEBUG);
<%}%>
<%if(isLog4j2Enabled) {%>
org.apache.logging.log4j.core.config.Configurator.setLevel(p_<%=jobCatcherNode.getUniqueName()%>.getProperty("root.logger"), org.apache.logging.log4j.Level.DEBUG);
<%}%>
runtime_lineage_logger_<%=jobCatcherNode.getUniqueName()%> = org.talend.job.audit.JobEventAuditLoggerFactory.createJobAuditLogger(p_<%=jobCatcherNode.getUniqueName()%>);
<%
}
if(enableLogStash) {
String location = ElementParameterParser.getValue(jobCatcherNode, "__LOCATION__");
%>
if(enableLogStash) {
java.util.Properties properties_<%=jobCatcherNode.getUniqueName()%> = new java.util.Properties();
@@ -519,21 +600,21 @@
properties_<%=jobCatcherNode.getUniqueName()%>.setProperty("host", "false");
System.getProperties().stringPropertyNames().stream()
.filter(it -> it.startsWith("monitoring.audit.logger.properties."))
.forEach(key -> properties_<%=jobCatcherNode.getUniqueName()%>.setProperty(key.substring("monitoring.audit.logger.properties.".length()), System.getProperty(key)));
.filter(it -> it.startsWith("audit.logger."))
.forEach(key -> properties_<%=jobCatcherNode.getUniqueName()%>.setProperty(key.substring("audit.logger.".length()), System.getProperty(key)));
<%if(isLog4j1Enabled) {%>
org.apache.log4j.Logger.getLogger("audit").setLevel(org.apache.log4j.Level.DEBUG);
org.apache.log4j.Logger.getLogger(properties_<%=jobCatcherNode.getUniqueName()%>.getProperty("root.logger")).setLevel(org.apache.log4j.Level.DEBUG);
<%}%>
<%if(isLog4j2Enabled) {%>
org.apache.logging.log4j.core.config.Configurator.setLevel("audit", org.apache.logging.log4j.Level.DEBUG);
org.apache.logging.log4j.core.config.Configurator.setLevel(properties_<%=jobCatcherNode.getUniqueName()%>.getProperty("root.logger"), org.apache.logging.log4j.Level.DEBUG);
<%}%>
auditLogger_<%=jobCatcherNode.getUniqueName()%> = org.talend.job.audit.JobEventAuditLoggerFactory.createJobAuditLogger(properties_<%=jobCatcherNode.getUniqueName()%>);
}
<%
}
%>
%>
if(clientHost == null) {
clientHost = defaultClientHost;
@@ -569,6 +650,16 @@
}
%>
boolean inOSGi = routines.system.BundleUtils.inOSGi();
if (inOSGi) {
java.util.Dictionary<String, Object> jobProperties = routines.system.BundleUtils.getJobProperties(jobName);
if (jobProperties != null && jobProperties.get("context") != null) {
contextStr = (String)jobProperties.get("context");
}
}
try {
//call job/subjob with an existing context, like: --context=production. if without this parameter, there will use the default context instead.
java.io.InputStream inContext = <%=className%>.class.getClassLoader().getResourceAsStream("<%=jobClassPackageFolder%>/contexts/" + contextStr + ".properties");
@@ -576,13 +667,15 @@
inContext = <%=className%>.class.getClassLoader().getResourceAsStream("config/contexts/" + contextStr + ".properties");
}
if (inContext != null) {
//defaultProps is in order to keep the original context value
if(context != null && context.isEmpty()) {
try {
//defaultProps is in order to keep the original context value
if(context != null && context.isEmpty()) {
defaultProps.load(inContext);
context = new ContextProperties(defaultProps);
}
} finally {
inContext.close();
}
inContext.close();
} else if (!isDefaultContext) {
//print info and job continue to run, for case: context_param is not empty.
System.err.println("Could not find the context " + contextStr);
@@ -619,7 +712,16 @@
} //endIf
%>
context.setContextType("<%=ctxParam.getName()%>", "<%=ctxParam.getType()%>");
if(context.getStringValue("<%=ctxParam.getName()%>") == null) {
context.<%=ctxParam.getName()%> = null;
} else {
<%//}B1 block%>
<%
String typeToGenerate ="String";
if( !(ctxParam.getType().equals("id_File") || ctxParam.getType().equals("id_Directory") ||ctxParam.getType().equals("id_List Of Value"))) {
typeToGenerate=JavaTypesManager.getTypeToGenerate(ctxParam.getType(),true);
}
if (ctxParam.getType().equals("id_Password")) {
%>
String pwd_<%=ctxParam.getName()%>_value = context.getProperty("<%=ctxParam.getName()%>");
@@ -637,42 +739,41 @@
}
}
<%
continue;
}
String typeToGenerate ="String";
if( !(ctxParam.getType().equals("id_File") || ctxParam.getType().equals("id_Directory") ||ctxParam.getType().equals("id_List Of Value"))) {
typeToGenerate=JavaTypesManager.getTypeToGenerate(ctxParam.getType(),true);
}
if(typeToGenerate.equals("java.util.Date")) {
} else if(typeToGenerate.equals("java.util.Date")) {
%>
try{
String context_<%=ctxParam.getName()%>_value = context.getProperty("<%=ctxParam.getName()%>");
if (context_<%=ctxParam.getName()%>_value == null){
context_<%=ctxParam.getName()%>_value = "";
}
int context_<%=ctxParam.getName()%>_pos = context_<%=ctxParam.getName()%>_value.indexOf(";");
String context_<%=ctxParam.getName()%>_pattern = "yyyy-MM-dd HH:mm:ss";
if(context_<%=ctxParam.getName()%>_pos > -1){
context_<%=ctxParam.getName()%>_pattern = context_<%=ctxParam.getName()%>_value.substring(0, context_<%=ctxParam.getName()%>_pos);
context_<%=ctxParam.getName()%>_value = context_<%=ctxParam.getName()%>_value.substring(context_<%=ctxParam.getName()%>_pos + 1);
}
try{
if (context_<%=ctxParam.getName()%>_value == null){
context_<%=ctxParam.getName()%>_value = "";
}
int context_<%=ctxParam.getName()%>_pos = context_<%=ctxParam.getName()%>_value.indexOf(";");
String context_<%=ctxParam.getName()%>_pattern = "yyyy-MM-dd HH:mm:ss";
if(context_<%=ctxParam.getName()%>_pos > -1){
context_<%=ctxParam.getName()%>_pattern = context_<%=ctxParam.getName()%>_value.substring(0, context_<%=ctxParam.getName()%>_pos);
context_<%=ctxParam.getName()%>_value = context_<%=ctxParam.getName()%>_value.substring(context_<%=ctxParam.getName()%>_pos + 1);
}
context.<%=ctxParam.getName()%>=(java.util.Date)(new java.text.SimpleDateFormat(context_<%=ctxParam.getName()%>_pattern).parse(context_<%=ctxParam.getName()%>_value));
context.<%=ctxParam.getName()%>=(java.util.Date)(new java.text.SimpleDateFormat(context_<%=ctxParam.getName()%>_pattern).parse(context_<%=ctxParam.getName()%>_value));
} catch(ParseException e) {
} catch(ParseException e) {
try { <% /*try to check if date passed as long also*/ %>
long context_<%=ctxParam.getName()%>_longValue = Long.valueOf(context_<%=ctxParam.getName()%>_value);
context.<%=ctxParam.getName()%> = new java.util.Date(context_<%=ctxParam.getName()%>_longValue);
} catch (NumberFormatException cantParseToLongException) {
<%
if (isLog4jEnabled) {
if (isLog4jEnabled) {
%>
log.warn(String.format("<%=warningMessageFormat %>", "<%=ctxParam.getName() %>", e.getMessage()));
log.warn(String.format("<%=warningMessageFormat %>", "<%=ctxParam.getName() %>", "Can't parse date string: " + e.getMessage() + " and long: " + cantParseToLongException.getMessage()));
<%
} else {
} else {
%>
System.err.println(String.format("<%=warningMessageFormat %>", "<%=ctxParam.getName() %>", e.getMessage()));
System.err.println(String.format("<%=warningMessageFormat %>", "<%=ctxParam.getName() %>", "Can't parse date string: " + e.getMessage() + " and long: " + cantParseToLongException.getMessage()));
<%
}
%>
context.<%=ctxParam.getName()%>=null;
}
}
%>
context.<%=ctxParam.getName()%>=null;
}
<%
} else if(typeToGenerate.equals("Object")||typeToGenerate.equals("String")||typeToGenerate.equals("java.lang.String")) {
%>
@@ -702,6 +803,10 @@
}
<%
}
%>
<%//{B1 block%>
}
<%
}
%>
} <%/*close last method*/%>
@@ -872,20 +977,6 @@ if(execStat) {
}
%>
<%if (hadoopClusterIds.size() > 1) {%>
System.err.println("Only one hadoop configuration is allowed in one job!");
<%}%>
<%
// run tHadoopConfManager component if there is it.
if(!hadoopConfNodes.isEmpty()) {
INode hadoopConfNode = hadoopConfNodes.get(0);
%>
<%=createCallProcess(hadoopConfNode, className, false) %>
<%
}
%>
<%
//2. load implicit contextload
if(implicit_Context != null) {
@@ -908,6 +999,21 @@ this.globalResumeTicket = true;//to run tPreJob
}
%>
<%if (hadoopClusterIds.size() > 1) {%>
System.err.println("Only one hadoop configuration is allowed in one job!");
<%}%>
<%
// run tHadoopConfManager component if there is it.
if(!hadoopConfNodes.isEmpty()) {
INode hadoopConfNode = hadoopConfNodes.get(0);
if (hadoopConfNode.isStart()) {
%>
<%=createCallProcess(hadoopConfNode, className, false) %>
<%
}
}
%>
<%
//4. flush the begin msg
@@ -921,7 +1027,7 @@ this.globalResumeTicket = true;//to run tPreJob
<%
}
if(jobCatcherNode!=null) {
if(enableLogStash) {
%>
if(enableLogStash) {
<%=jobCatcherNode.getUniqueName() %>.addJobStartMessage();
@@ -1074,7 +1180,7 @@ this.globalResumeTicket = true;//to run tPostJob
<%
}
if(jobCatcherNode!=null) {
if(enableLogStash) {
%>
if(enableLogStash) {
<%=jobCatcherNode.getUniqueName() %>.addJobEndMessage(startTime, end, status);
@@ -1092,9 +1198,12 @@ this.globalResumeTicket = true;//to run tPostJob
closeJmsConnections();
<% } %>
<% if (exist_tSCP) { %>
closeScpConnections();
<% } %>
<% if (exist_tSCP) {
%>
closeCloseableConnections(<%=parameterNames%>);
<%
}
%>
<%
if (stats) {
@@ -1113,6 +1222,24 @@ if (execStat) {
}
%>
int returnCode = 0;
<%
if (isRunInMultiThread) {
%>
Integer localErrorCode = (Integer)(((java.util.Map)threadLocal.get()).get("errorCode"));
String localStatus = (String)(((java.util.Map)threadLocal.get()).get("status"));
if (localErrorCode != null) {
if (errorCode == null || localErrorCode.compareTo(errorCode) > 0) {
errorCode = localErrorCode;
}
}
if (localStatus != null && !status.equals("failure")){
status = localStatus;
}
<%
}
%>
if(errorCode == null) {
returnCode = status != null && status.equals("failure") ? 1 : 0;
} else {
@@ -1130,7 +1257,7 @@ if (execStat) {
closeJmsConnections();
<% } %>
<% if(exist_tSCP) { %>
closeScpConnections();
closeCloseableConnections(<%=parameterNames%>);
<% } %>
<% if (exist_tSQLDB) { %>
closeSqlDbConnections();
@@ -1198,22 +1325,17 @@ if (execStat) {
<%
if(exist_tSCP) {
%>
private void closeScpConnections() {
try {
Object obj_conn;
<%
for (INode scpNode : scpComponentsList) {
%>
obj_conn = globalMap.remove("conn_<%=scpNode.getUniqueName() %>");
if (null != obj_conn) {
((ch.ethz.ssh2.Connection) obj_conn).close();
private void closeCloseableConnections(String... names) {
java.util.Arrays.stream(names).forEach(name-> {
try {
Object obj_conn = globalMap.remove(name);
if(obj_conn != null){
((java.io.Closeable)obj_conn).close();
}
} catch (IOException ioException) {
}
<%
}
%>
} catch (java.lang.Exception e) {
}
}
});
}
<%
}
%>
@@ -1320,13 +1442,12 @@ if (execStat) {
} else {
type = "ftp";
}
if ("ftp".equals(type)) { %>
((com.enterprisedt.net.ftp.FTPClient) obj_conn).quit();
<%} else if ("sftp".equals(type)) { %>
if ("sftp".equals(type)) { %>
((com.jcraft.jsch.ChannelSftp) obj_conn).quit();
<%} else { %>
((org.apache.commons.net.ftp.FTPSClient) obj_conn).disconnect();
<%}%>
<%} else { %>
((org.apache.commons.net.ftp.FTPClient) obj_conn).logout();
((org.apache.commons.net.ftp.FTPClient) obj_conn).disconnect();
<%}%>
}
<% } %>
} catch (java.lang.Exception e) {
@@ -1416,11 +1537,24 @@ if (execStat) {
private void closeGSConnections() {
try {
<% for (INode gsComponent : gsComponentsList) { %>
org.jets3t.service.impl.rest.httpclient.GoogleStorageService service_<%=gsComponent.getUniqueName()%> =
(org.jets3t.service.impl.rest.httpclient.GoogleStorageService)globalMap.get("service_<%=gsComponent.getUniqueName()%>");
if(service_<%=gsComponent.getUniqueName()%> !=null){
service_<%=gsComponent.getUniqueName()%>.shutdown();
}
<%
String authType = ElementParameterParser.getValue(gsComponent, "__AUTH_TYPE__");
if (authType.equals("USER_ACCOUNT_HMAC")){
%>
org.jets3t.service.impl.rest.httpclient.GoogleStorageService service_<%=gsComponent.getUniqueName()%> =
(org.jets3t.service.impl.rest.httpclient.GoogleStorageService)globalMap.get("service_<%=gsComponent.getUniqueName()%>");
if(service_<%=gsComponent.getUniqueName()%> !=null){
service_<%=gsComponent.getUniqueName()%>.shutdown();
}
<%
} else if (authType.equals("SERVICE_ACCOUNT")) {
%>
//Don't need to close storage when use google storage api
<%
}
%>
<% } %>
} catch (java.lang.Exception e) {
@@ -1501,12 +1635,14 @@ if (execStat) {
<% } %>
<% for (INode existNode : existComponentsList) { %>
connections.put("col_<%=existNode.getUniqueName()%>", globalMap.get("col_<%=existNode.getUniqueName()%>"));
<% } %>
<% }
<% for (INode salesforceNode : salesforceComponentsList) { %>
connections.put("conn_<%=salesforceNode.getUniqueName()%>", globalMap.get("conn_<%=salesforceNode.getUniqueName()%>"));
<% } %>
<% for (INode sapNode : sapComponentsList) { %>
for (INode tcompV0ConnectionNode : tcompV0ComponentsList) { %>
connections.put("<%=tcompV0ConnectionNode.getUniqueName()%>_connection", globalMap.get("<%=tcompV0ConnectionNode.getUniqueName()%>_connection"));
connections.put("<%=tcompV0ConnectionNode.getUniqueName()%>_COMPONENT_RUNTIME_PROPERTIES", globalMap.get("<%=tcompV0ConnectionNode.getUniqueName()%>_COMPONENT_RUNTIME_PROPERTIES"));
<% }
for (INode sapNode : sapComponentsList) { %>
connections.put("conn_<%=sapNode.getUniqueName()%>", globalMap.get("conn_<%=sapNode.getUniqueName()%>"));
<% } %>
<% for (INode gsNode : gsComponentsList) { %>
@@ -1570,7 +1706,7 @@ if (execStat) {
}
} else if (arg.startsWith("--log4jLevel=")) {
log4jLevel = arg.substring(13);
} else if (arg.startsWith("--monitoring") && arg.contains("=")) {//for trunjob call
} else if (arg.startsWith("--audit.enabled") && arg.contains("=")) {//for trunjob call
final int equal = arg.indexOf('=');
final String key = arg.substring("--".length(), equal);
System.setProperty(key, arg.substring(equal + 1));

View File

@@ -56,10 +56,28 @@ if ((metadatas != null) && (metadatas.size() > 0)) { // metadata
// Set up the component definition, and the properties for all types of
// components.
List<? extends IConnection> allInLineJobConns = NodeUtil.getFirstIncomingLineConnectionsOfType(node, "tRESTRequestIn");
%>
boolean doesNodeBelongToRequest_<%=cid%> = <%= allInLineJobConns.size() %> == 0;
@SuppressWarnings("unchecked")
java.util.Map<String, Object> restRequest_<%=cid%> = (java.util.Map<String, Object>)globalMap.get("restRequest");
String currentTRestRequestOperation_<%=cid%> = (String)(restRequest_<%=cid%> != null ? restRequest_<%=cid%>.get("OPERATION") : null);
<%
for (IConnection inLineConn : allInLineJobConns) {
%>
if("<%= inLineConn.getName() %>".equals(currentTRestRequestOperation_<%=cid%>)) {
doesNodeBelongToRequest_<%=cid%> = true;
}
<%
}
%>
org.talend.components.api.component.ComponentDefinition def_<%=cid %> =
new <%= def.getClass().getName()%>();
org.talend.components.api.component.runtime.Writer writer_<%=cid%> = null;
org.talend.components.api.component.runtime.Reader reader_<%=cid%> = null;
<%
List<Component.CodegenPropInfo> propsToProcess = component.getCodegenPropInfos(componentProps);
%>
@@ -145,7 +163,7 @@ globalMap.put("TALEND_COMPONENTS_VERSION", "<%=component.getVersion()%>");
boolean isParallelize ="true".equalsIgnoreCase(ElementParameterParser.getValue(node, "__PARALLELIZE__"));
if (isParallelize) {
%>
final String buffersSizeKey_<%=cid%> = "buffersSizeKey_<%=cid%>_" + Thread.currentThread().getId();
final String buffersSizeKey_<%=cid%> = "buffersSizeKey_<%=cid%>_" + Thread.currentThread().getId();
<%
}
%>
@@ -215,9 +233,11 @@ if(componentRuntime_<%=cid%> instanceof org.talend.components.api.component.runt
org.talend.components.api.component.runtime.SourceOrSink sourceOrSink_<%=cid%> = null;
if(componentRuntime_<%=cid%> instanceof org.talend.components.api.component.runtime.SourceOrSink) {
sourceOrSink_<%=cid%> = (org.talend.components.api.component.runtime.SourceOrSink)componentRuntime_<%=cid%>;
org.talend.daikon.properties.ValidationResult vr_<%=cid%> = sourceOrSink_<%=cid%>.validate(container_<%=cid%>);
if (vr_<%=cid%>.getStatus() == org.talend.daikon.properties.ValidationResult.Result.ERROR ) {
throw new RuntimeException(vr_<%=cid%>.getMessage());
if (doesNodeBelongToRequest_<%=cid%>) {
org.talend.daikon.properties.ValidationResult vr_<%=cid%> = sourceOrSink_<%=cid%>.validate(container_<%=cid%>);
if (vr_<%=cid%>.getStatus() == org.talend.daikon.properties.ValidationResult.Result.ERROR ) {
throw new RuntimeException(vr_<%=cid%>.getMessage());
}
}
}
@@ -240,11 +260,11 @@ if(isTopologyNone) {
if (hasOutputOnly || asInputComponent) {
%>
org.talend.components.api.component.runtime.Source source_<%=cid%> =
(org.talend.components.api.component.runtime.Source)sourceOrSink_<%=cid%>;
org.talend.components.api.component.runtime.Reader reader_<%=cid%> =
source_<%=cid%>.createReader(container_<%=cid%>);
reader_<%=cid%> = new org.talend.codegen.flowvariables.runtime.FlowVariablesReader(reader_<%=cid%>, container_<%=cid%>);
if (sourceOrSink_<%=cid%> instanceof org.talend.components.api.component.runtime.Source) {
org.talend.components.api.component.runtime.Source source_<%=cid%> =
(org.talend.components.api.component.runtime.Source)sourceOrSink_<%=cid%>;
reader_<%=cid%> = source_<%=cid%>.createReader(container_<%=cid%>);
reader_<%=cid%> = new org.talend.codegen.flowvariables.runtime.FlowVariablesReader(reader_<%=cid%>, container_<%=cid%>);
<%
IConnection main = null;
@@ -266,19 +286,19 @@ if (hasOutputOnly || asInputComponent) {
IConnection schemaSourceConnector = (main!=null ? main : reject);
String schemaSourceConnectorName = schemaSourceConnector.getMetadataTable().getAttachedConnector();
%>
boolean multi_output_is_allowed_<%=cid%> = false;
boolean multi_output_is_allowed_<%=cid%> = false;
<% //take care SourceOrSink.validate will change the schema if it contains include-all-fields, so need to get design Avro schema before validate %>
org.talend.components.api.component.Connector c_<%=cid%> = null;
for (org.talend.components.api.component.Connector currentConnector : props_<%=cid %>.getAvailableConnectors(null, true)) {
if (currentConnector.getName().equals("<%=schemaSourceConnectorName%>")) {
c_<%=cid%> = currentConnector;
}
org.talend.components.api.component.Connector c_<%=cid%> = null;
for (org.talend.components.api.component.Connector currentConnector : props_<%=cid %>.getAvailableConnectors(null, true)) {
if (currentConnector.getName().equals("<%=schemaSourceConnectorName%>")) {
c_<%=cid%> = currentConnector;
}
if (currentConnector.getName().equals("REJECT")) {//it's better to move the code to javajet
multi_output_is_allowed_<%=cid%> = true;
if (currentConnector.getName().equals("REJECT")) {//it's better to move the code to javajet
multi_output_is_allowed_<%=cid%> = true;
}
}
}
org.apache.avro.Schema schema_<%=cid%> = props_<%=cid %>.getSchema(c_<%=cid%>, true);
org.apache.avro.Schema schema_<%=cid%> = props_<%=cid %>.getSchema(c_<%=cid%>, true);
<%
irToRow = new IndexedRecordToRowStructGenerator(cid, null, columnList);
@@ -286,117 +306,123 @@ if (hasOutputOnly || asInputComponent) {
}
%>
// Iterate through the incoming data.
boolean available_<%=cid%> = reader_<%=cid%>.start();
// Iterate through the incoming data.
boolean available_<%=cid%> = reader_<%=cid%>.start();
resourceMap.put("reader_<%=cid%>", reader_<%=cid%>);
resourceMap.put("reader_<%=cid%>", reader_<%=cid%>);
for (; available_<%=cid%>; available_<%=cid%> = reader_<%=cid%>.advance()) {
nb_line_<%=cid %>++;
for (; available_<%=cid%>; available_<%=cid%> = reader_<%=cid%>.advance()) {
nb_line_<%=cid %>++;
<%if(hasDataOutput) {%>
if (multi_output_is_allowed_<%=cid%>) {
<%if(main!=null){%>
<%=main.getName()%> = null;
<%}%>
<%if(hasDataOutput) {%>
if (multi_output_is_allowed_<%=cid%>) {
<%if(main!=null){%>
<%=main.getName()%> = null;
<%}%>
<%if(reject!=null){%>
<%=reject.getName()%> = null;
<%}%>
}
<%}%>
<%if(reject!=null){%>
<%=reject.getName()%> = null;
<%}%>
}
<%}%>
try {
Object data_<%=cid%> = reader_<%=cid%>.getCurrent();
<%
if (main != null) {
%>
try {
Object data_<%=cid%> = reader_<%=cid%>.getCurrent();
<%
if (main != null) {
%>
if(multi_output_is_allowed_<%=cid%>) {
<%=main.getName()%> = new <%=main.getName() %>Struct();
}
if(multi_output_is_allowed_<%=cid%>) {
<%=main.getName()%> = new <%=main.getName() %>Struct();
}
<%
irToRow.generateConvertRecord("data_" + cid, main.getName(), main.getMetadataTable().getListColumns());
}
%>
} catch (org.talend.components.api.exception.DataRejectException e_<%=cid%>) {
java.util.Map<String,Object> info_<%=cid%> = e_<%=cid%>.getRejectInfo();
<%
if (reject!=null) {
%>
Object data_<%=cid%> = info_<%=cid%>.get("talend_record");
<%
irToRow.generateConvertRecord("data_" + cid, main.getName(), main.getMetadataTable().getListColumns());
}
%>
} catch (org.talend.components.api.exception.DataRejectException e_<%=cid%>) {
java.util.Map<String,Object> info_<%=cid%> = e_<%=cid%>.getRejectInfo();
<%
if (reject!=null) {
%>
Object data_<%=cid%> = info_<%=cid%>.get("talend_record");
if (multi_output_is_allowed_<%=cid%>) {
<%=reject.getName()%> = new <%=reject.getName() %>Struct();
}
try{
<%
irToRow.generateConvertRecord("data_" + cid, reject.getName());
%>
}catch(java.lang.Exception e){
// do nothing
}
<%
Set<String> commonColumns = new HashSet<String>();
if (multi_output_is_allowed_<%=cid%>) {
<%=reject.getName()%> = new <%=reject.getName() %>Struct();
}
try{
<%
irToRow.generateConvertRecord("data_" + cid, reject.getName());
%>
}catch(java.lang.Exception e){
// do nothing
}
<%
Set<String> commonColumns = new HashSet<String>();
for (IMetadataColumn column : columnList) {
commonColumns.add(column.getLabel());
}
for (IMetadataColumn column : columnList) {
commonColumns.add(column.getLabel());
}
//pass error columns
List<IMetadataColumn> rejectColumns = reject.getMetadataTable().getListColumns();
for(IMetadataColumn column : rejectColumns) {
String columnName = column.getLabel();
//pass error columns
List<IMetadataColumn> rejectColumns = reject.getMetadataTable().getListColumns();
for(IMetadataColumn column : rejectColumns) {
String columnName = column.getLabel();
// JavaType javaType = JavaTypesManager.getJavaTypeFromId(column.getTalendType());
String typeToGenerate = JavaTypesManager.getTypeToGenerate(column.getTalendType(), column.isNullable());
// JavaType javaType = JavaTypesManager.getJavaTypeFromId(column.getTalendType());
String typeToGenerate = JavaTypesManager.getTypeToGenerate(column.getTalendType(), column.isNullable());
//error columns
if(!commonColumns.contains(columnName)) {
%>
<%=reject.getName()%>.<%=columnName%> = (<%=typeToGenerate%>)info_<%=cid%>.get("<%=columnName%>");
<%
}
}
} else {
%>
//TODO use a method instead of getting method by the special key "error/errorMessage"
Object errorMessage_<%=cid%> = null;
if(info_<%=cid%>.containsKey("error")){
errorMessage_<%=cid%> = info_<%=cid%>.get("error");
}else if(info_<%=cid%>.containsKey("errorMessage")){
errorMessage_<%=cid%> = info_<%=cid%>.get("errorMessage");
}else{
errorMessage_<%=cid%> = "Rejected but error message missing";
}
errorMessage_<%=cid%> = "Row "+ nb_line_<%=cid %> + ": "+errorMessage_<%=cid%>;
System.err.println(errorMessage_<%=cid%>);
<%
}
if (main != null) {
%>
// If the record is reject, the main line record should put NULL
<%=main.getName()%> = null;
<%
}
%>
}
//error columns
if(!commonColumns.contains(columnName)) {
%>
<%=reject.getName()%>.<%=columnName%> = (<%=typeToGenerate%>)info_<%=cid%>.get("<%=columnName%>");
<%
}
}
} else {
%>
//TODO use a method instead of getting method by the special key "error/errorMessage"
Object errorMessage_<%=cid%> = null;
if(info_<%=cid%>.containsKey("error")){
errorMessage_<%=cid%> = info_<%=cid%>.get("error");
}else if(info_<%=cid%>.containsKey("errorMessage")){
errorMessage_<%=cid%> = info_<%=cid%>.get("errorMessage");
}else{
errorMessage_<%=cid%> = "Rejected but error message missing";
}
errorMessage_<%=cid%> = "Row "+ nb_line_<%=cid %> + ": "+errorMessage_<%=cid%>;
System.err.println(errorMessage_<%=cid%>);
<%
}
if (main != null) {
%>
// If the record is reject, the main line record should put NULL
<%=main.getName()%> = null;
<%
}
%>
} // end of catch
<%
// The for loop around the incoming records from the reader is left open.
} else if (hasInput) {
%>
org.talend.components.api.component.runtime.Sink sink_<%=cid%> =
(org.talend.components.api.component.runtime.Sink)sourceOrSink_<%=cid%>;
org.talend.components.api.component.runtime.WriteOperation writeOperation_<%=cid%> = sink_<%=cid%>.createWriteOperation();
writeOperation_<%=cid%>.initialize(container_<%=cid%>);
org.talend.components.api.component.runtime.Writer writer_<%=cid%> = writeOperation_<%=cid%>.createWriter(container_<%=cid%>);
writer_<%=cid%>.open("<%=cid%>");
resourceMap.put("writer_<%=cid%>", writer_<%=cid%>);
org.talend.codegen.enforcer.IncomingSchemaEnforcer incomingEnforcer_<%=cid%> = null;
if (sourceOrSink_<%=cid%> instanceof org.talend.components.api.component.runtime.Sink) {
org.talend.components.api.component.runtime.Sink sink_<%=cid%> =
(org.talend.components.api.component.runtime.Sink)sourceOrSink_<%=cid%>;
org.talend.components.api.component.runtime.WriteOperation writeOperation_<%=cid%> = sink_<%=cid%>.createWriteOperation();
if (doesNodeBelongToRequest_<%=cid%>) {
writeOperation_<%=cid%>.initialize(container_<%=cid%>);
}
writer_<%=cid%> = writeOperation_<%=cid%>.createWriter(container_<%=cid%>);
if (doesNodeBelongToRequest_<%=cid%>) {
writer_<%=cid%>.open("<%=cid%>");
}
resourceMap.put("writer_<%=cid%>", writer_<%=cid%>);
} // end of "sourceOrSink_<%=cid%> instanceof ...Sink"
org.talend.components.api.component.Connector c_<%=cid%> = null;
for (org.talend.components.api.component.Connector currentConnector : props_<%=cid %>.getAvailableConnectors(null, false)) {
if (currentConnector.getName().equals("MAIN")) {
@@ -405,8 +431,7 @@ if (hasOutputOnly || asInputComponent) {
}
}
org.apache.avro.Schema designSchema_<%=cid%> = props_<%=cid %>.getSchema(c_<%=cid%>, false);
org.talend.codegen.enforcer.IncomingSchemaEnforcer incomingEnforcer_<%=cid%>
= new org.talend.codegen.enforcer.IncomingSchemaEnforcer(designSchema_<%=cid%>);
incomingEnforcer_<%=cid%> = new org.talend.codegen.enforcer.IncomingSchemaEnforcer(designSchema_<%=cid%>);
<%
List<? extends IConnection> outgoingConns = node.getOutgoingSortedConnections();
if (outgoingConns!=null){
@@ -442,7 +467,8 @@ if (hasOutputOnly || asInputComponent) {
}
}
}
%>
%>
java.lang.Iterable<?> outgoingMainRecordsList_<%=cid%> = new java.util.ArrayList<Object>();
java.util.Iterator outgoingMainRecordsIt_<%=cid%> = null;

View File

@@ -58,13 +58,24 @@ if(isTopologyNone) {
else if(hasOutputOnly || asInputComponent){
%>
} // while
reader_<%=cid%>.close();
final java.util.Map<String, Object> resultMap_<%=cid%> = reader_<%=cid%>.getReturnValues();
<%
if (hasOutputOnly || asInputComponent) {
%>
} // end of "if (sourceOrSink_<%=cid%> instanceof ...Source)"
<% } %>
java.util.Map<String, Object> resultMap_<%=cid%> = null;
if (reader_<%=cid%> != null) {
reader_<%=cid%>.close();
resultMap_<%=cid%> = reader_<%=cid%>.getReturnValues();
}
<%
}else if(hasInput){
%>
org.talend.components.api.component.runtime.Result resultObject_<%=cid%> = (org.talend.components.api.component.runtime.Result)writer_<%=cid%>.close();
final java.util.Map<String, Object> resultMap_<%=cid%> = writer_<%=cid%>.getWriteOperation().finalize(java.util.Arrays.<org.talend.components.api.component.runtime.Result>asList(resultObject_<%=cid%>), container_<%=cid%>);
java.util.Map<String, Object> resultMap_<%=cid%> = null;
if (writer_<%=cid%> != null) {
org.talend.components.api.component.runtime.Result resultObject_<%=cid%> = (org.talend.components.api.component.runtime.Result)writer_<%=cid%>.close();
resultMap_<%=cid%> = writer_<%=cid%>.getWriteOperation().finalize(java.util.Arrays.<org.talend.components.api.component.runtime.Result>asList(resultObject_<%=cid%>), container_<%=cid%>);
}
<%
} else {
return stringBuffer.toString();

View File

@@ -84,7 +84,7 @@ if(hasInput){
for (int i = 0; i < input_columnList.size(); i++) {
if(!input_columnList.get(i).getTalendType().equals("id_Dynamic")) {
%>
if (incomingEnforcer_<%=cid%>.getDesignSchema().getField("<%=input_columnList.get(i)%>") == null){
if (incomingEnforcer_<%=cid%> != null && incomingEnforcer_<%=cid%>.getDesignSchema().getField("<%=input_columnList.get(i)%>") == null){
incomingEnforcer_<%=cid%>.addIncomingNodeField("<%=input_columnList.get(i)%>", ((Object) <%=inputConn.getName()%>.<%=input_columnList.get(i)%>).getClass().getCanonicalName());
shouldCreateRuntimeSchemaForIncomingNode = true;
}
@@ -92,7 +92,7 @@ if(hasInput){
}
}
%>
if (shouldCreateRuntimeSchemaForIncomingNode){
if (shouldCreateRuntimeSchemaForIncomingNode && incomingEnforcer_<%=cid%> != null){
incomingEnforcer_<%=cid%>.createRuntimeSchema();
}
<%
@@ -111,7 +111,7 @@ if(hasInput){
if (dynamicPos != -1) {
%>
if (!incomingEnforcer_<%=cid%>.areDynamicFieldsInitialized()) {
if (incomingEnforcer_<%=cid%> != null && !incomingEnforcer_<%=cid%>.areDynamicFieldsInitialized()) {
// Initialize the dynamic columns when they are first encountered.
for (routines.system.DynamicMetadata dm_<%=cid%> : <%=inputConn.getName()%>.<%=input_columnList.get(dynamicPos).getLabel()%>.metadatas) {
incomingEnforcer_<%=cid%>.addDynamicField(
@@ -120,7 +120,8 @@ if(hasInput){
dm_<%=cid%>.getLogicalType(),
dm_<%=cid%>.getFormat(),
dm_<%=cid%>.getDescription(),
dm_<%=cid%>.isNullable());
dm_<%=cid%>.isNullable(),
dm_<%=cid%>.isKey());
}
incomingEnforcer_<%=cid%>.createRuntimeSchema();
}
@@ -128,22 +129,26 @@ if(hasInput){
}
%>
incomingEnforcer_<%=cid%>.createNewRecord();
if (incomingEnforcer_<%=cid%> != null) {
incomingEnforcer_<%=cid%>.createNewRecord();
}
<%
for (int i = 0; i < input_columnList.size(); i++) { // column
IMetadataColumn column = input_columnList.get(i);
if (dynamicPos != i) {
%>
//skip the put action if the input column doesn't appear in component runtime schema
if (incomingEnforcer_<%=cid%>.getRuntimeSchema().getField("<%=input_columnList.get(i)%>") != null){
if (incomingEnforcer_<%=cid%> != null && incomingEnforcer_<%=cid%>.getRuntimeSchema().getField("<%=input_columnList.get(i)%>") != null){
incomingEnforcer_<%=cid%>.put("<%=column.getLabel()%>", <%=inputConn.getName()%>.<%=column.getLabel()%>);
}
<%
} else {
%>
for (int i = 0; i < <%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnCount(); i++) {
incomingEnforcer_<%=cid%>.put(<%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnMetadata(i).getName(),
<%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnValue(i));
if (incomingEnforcer_<%=cid%> != null) {
for (int i = 0; i < <%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnCount(); i++) {
incomingEnforcer_<%=cid%>.put(<%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnMetadata(i).getName(),
<%=inputConn.getName()%>.<%=column.getLabel()%>.getColumnValue(i));
}
}
<%
}
@@ -177,7 +182,11 @@ if(hasInput){
} // propInfo
%>
org.apache.avro.generic.IndexedRecord data_<%=cid%> = incomingEnforcer_<%=cid%>.getCurrentRecord();
org.apache.avro.generic.IndexedRecord data_<%=cid%> = null;
if (incomingEnforcer_<%=cid%> != null) {
data_<%=cid%> = incomingEnforcer_<%=cid%>.getCurrentRecord();
}
<%
boolean isParallelize ="true".equalsIgnoreCase(ElementParameterParser.getValue(node, "__PARALLELIZE__"));
@@ -190,8 +199,9 @@ if(hasInput){
}
}
%>
writer_<%=cid%>.write(data_<%=cid%>);
if (writer_<%=cid%> != null && data_<%=cid%> != null) {
writer_<%=cid%>.write(data_<%=cid%>);
}
nb_line_<%=cid %>++;
<%if(hasMainOutput){

View File

@@ -116,7 +116,11 @@ class IndexedRecordToRowStructGenerator {
routines.system.DynamicMetadata dynamicMetadata_<%=cid%> = new routines.system.DynamicMetadata();
org.apache.avro.Schema dynamicFieldSchema_<%=cid%> = dynamicField_<%=cid%>.schema();
// set name
dynamicMetadata_<%=cid%>.setName(dynamicField_<%=cid%>.name());
if("true".equals(dynamicField_<%=cid%>.getProp("ENABLE_SPECIAL_TABLENAME"))){
dynamicMetadata_<%=cid%>.setName(dynamicField_<%=cid%>.getProp("talend.field.dbColumnName"));
}else{
dynamicMetadata_<%=cid%>.setName(dynamicField_<%=cid%>.name());
}
// set db name
dynamicMetadata_<%=cid%>.setDbName(dynamicField_<%=cid%>.name());
// set nullable
@@ -166,6 +170,14 @@ class IndexedRecordToRowStructGenerator {
if (columnName.equals(dynamicColName)) {
%>
java.util.Map<String, Object> dynamicValue_<%=cid%> = (java.util.Map<String, Object>) <%=codeVarSchemaEnforcer%>.get(<%=i%>);
org.apache.avro.Schema dynSchema_<%=cid%> = ((org.talend.codegen.enforcer.OutgoingDynamicSchemaEnforcer) <%=codeVarSchemaEnforcer%>).getDynamicFieldsSchema();
for (org.apache.avro.Schema.Field dynamicField_<%=cid%> : dynSchema_<%=cid%>.getFields()){
String name = dynamicField_<%=cid%>.name();
if("true".equals(dynamicField_<%=cid%>.getProp("ENABLE_SPECIAL_TABLENAME"))){
dynamicValue_<%=cid%>.put(dynamicField_<%=cid%>.getProp("talend.field.dbColumnName"), dynamicValue_<%=cid%>.get(name));
dynamicValue_<%=cid%>.remove(name);
}
}
for (java.util.Map.Entry<String, Object> dynamicValueEntry_<%=cid%> : dynamicValue_<%=cid%>.entrySet()) {
<%=codeVarDynamic%>.setColumnValue(<%=codeVarDynamic%>.getIndex(dynamicValueEntry_<%=cid%>.getKey()), dynamicValueEntry_<%=cid%>.getValue());
}

View File

@@ -73,6 +73,9 @@ import pigudf.<%=routine%>;
import routines.<%=routine%>;
<% }
}%>
<%for (String codesJar : CodeGeneratorRoutine.getRequiredCodesJarName(process)) {%>
import <%=codesJar%>;
<%}%>
import routines.system.*;
import routines.system.api.*;
import java.text.ParseException;
@@ -298,6 +301,15 @@ if(isRunInMultiThread){
}
<% } %>
}
//if the stored or passed value is "<TALEND_NULL>" string, it mean null
public String getStringValue(String key) {
String origin_value = this.getProperty(key);
if(NULL_VALUE_EXPRESSION_IN_COMMAND_STRING_FOR_CHILD_JOB_ONLY.equals(origin_value)) {
return null;
}
return origin_value;
}
<%
for (IContextParameter ctxParam :params)
@@ -380,8 +392,28 @@ public <%=JavaTypesManager.getTypeToGenerate(ctxParam.getType(),true)%> get<%=Ch
break;
}
}
boolean enableLogStash = jobCatcherNode != null;
if (stats || enableLogStash) {
boolean enableLogStash = !Boolean.getBoolean("deactivate_extended_component_log") && (jobCatcherNode!=null);
boolean enable_runtime_lineage_log = NodeUtil.isJobUsingRuntimeLineage(process) && (jobCatcherNode!=null);
if(jobCatcherNode!=null) {
%>
private final JobStructureCatcherUtils <%=jobCatcherNode.getUniqueName() %> = new JobStructureCatcherUtils(jobName, "<%=process.getId() %>", "<%=process.getVersion() %>");
<%
}
if(enable_runtime_lineage_log) {
%>
private org.talend.job.audit.JobAuditLogger runtime_lineage_logger_<%=jobCatcherNode.getUniqueName()%> = null;
<%
}
if (enableLogStash) {
%>
private org.talend.job.audit.JobAuditLogger auditLogger_<%=jobCatcherNode.getUniqueName()%> = null;
private RunStat runStat = new RunStat(<%=jobCatcherNode.getUniqueName() %>, System.getProperty("audit.interval"));
<%
} else if(stats) {
%>
private RunStat runStat = new RunStat();
<%
@@ -408,6 +440,20 @@ private RunTrace runTrace = new RunTrace();
globalMap.put(KEY_DB_DATASOURCES, talendDataSources);
globalMap.put(KEY_DB_DATASOURCES_RAW, new java.util.HashMap<String, javax.sql.DataSource>(dataSources));
}
public void setDataSourceReferences(List serviceReferences) throws Exception{
java.util.Map<String, routines.system.TalendDataSource> talendDataSources = new java.util.HashMap<String, routines.system.TalendDataSource>();
java.util.Map<String, javax.sql.DataSource> dataSources = new java.util.HashMap<String, javax.sql.DataSource>();
for (java.util.Map.Entry<String, javax.sql.DataSource> entry : BundleUtils.getServices(serviceReferences, javax.sql.DataSource.class).entrySet()) {
dataSources.put(entry.getKey(), entry.getValue());
talendDataSources.put(entry.getKey(), new routines.system.TalendDataSource(entry.getValue()));
}
globalMap.put(KEY_DB_DATASOURCES, talendDataSources);
globalMap.put(KEY_DB_DATASOURCES_RAW, new java.util.HashMap<String, javax.sql.DataSource>(dataSources));
}
<%
for (INode logCatcher : process.getNodesOfType("tLogCatcher")) {
@@ -422,13 +468,6 @@ private RunTrace runTrace = new RunTrace();
<%
}
if(enableLogStash) {
%>
JobStructureCatcherUtils <%=jobCatcherNode.getUniqueName() %> = new JobStructureCatcherUtils(jobName, "<%=process.getId() %>", "<%=process.getVersion() %>");
org.talend.job.audit.JobAuditLogger auditLogger_<%=jobCatcherNode.getUniqueName()%> = null;
<%
}
for (INode metterCatcher : process.getNodesOfType("tFlowMeterCatcher")) {
%>
MetterCatcherUtils <%=metterCatcher.getUniqueName() %> = new MetterCatcherUtils("<%=process.getId() %>", "<%=process.getVersion() %>");

View File

@@ -12,6 +12,7 @@
org.talend.designer.codegen.config.CodeGeneratorArgument
org.talend.designer.codegen.config.NodesSubTree
org.talend.core.model.process.IProcess
org.talend.core.model.process.ProcessUtils
org.talend.core.model.utils.NodeUtil
org.talend.core.model.process.IContextParameter
java.util.List
@@ -28,7 +29,13 @@ INode node = (INode)codeGenArgument.getArgument();
boolean containsTPartitioner = node.getProcess().getNodesOfType("tPartitioner").size() > 0 ? true : false;
boolean isRunJob = "tRunJob".equals(node.getComponent().getName());
IProcess process = node.getProcess();
boolean isTestContainer=ProcessUtils.isTestContainer(process);
String className = process.getName();
if (isTestContainer) {
className = className + "Test";
}
NodesSubTree subTree = (NodesSubTree) codeGenArgument.getSubTree();
ECodePart codePart = codeGenArgument.getCodePart();
//boolean trace = codeGenArgument.isTrace();
@@ -78,10 +85,10 @@ for (IConnection iterateConn : iterateConnSet) { //1
@Override
public Object put(String key, Object value) {
<%if(!isRunInMultiThread){%>
synchronized (<%=process.getName()%>.this.obj) {
synchronized (<%=className%>.this.obj) {
<%}%>
super.put(key, value);
return <%=process.getName()%>.this.globalMap.put(key, value);
return <%=className%>.this.globalMap.put(key, value);
<%if(!isRunInMultiThread){%>
}
<%}%>
@@ -158,7 +165,7 @@ for (IConnection iterateConn : iterateConnSet) { //1
synchronized (globalMap) {
this.globalMap = java.util.Collections.synchronizedMap(new ThreadedMap(globalMap));
<%}else{%>
synchronized (<%=process.getName()%>.this.obj) {
synchronized (<%=className%>.this.obj) {
this.globalMap = new ThreadedMap(globalMap);
<%}%>
}

View File

@@ -44,7 +44,7 @@
INode startNode = subTree.getRootNode();
String startNodeId = startNode.getUniqueName();
if(startNodeId!=null && startNodeId.startsWith("tCollector")) {
if ("tCollector".equals( startNode.getComponent().getOriginalName() )) {
List<? extends INode> departitioners = startNode.getProcess().getNodesOfType("tDepartitioner");
if(departitioners!=null) {
for(INode departitioner : departitioners) {

View File

@@ -0,0 +1,121 @@
<%
//copy from configuration.javajet for tacokit
%>
<%
//TODO: modify this part for Maps and nested lists.
if (p.getFieldType() == EParameterFieldType.TABLE || p.getFieldType() == EParameterFieldType.TACOKIT_SUGGESTABLE_TABLE) {
java.util.List<java.util.Map<String, String>> tableValues = ElementParameterParser.createTableValues((java.util.List<java.util.Map<String, Object>>) p.getValue(), p);
String[] items = p.getListItemsDisplayCodeName();
String tableName = p.getName().replace('$', '.');
boolean primitiveTable = items.length == 1 && items[0].equals(tableName + "[]");
String tableNamePrefix = tableName + "[]";
for (int i = 0; i < tableValues.size(); i++) {
java.util.Map<String, String> lineValues = tableValues.get(i);
for (int j = 0; j < items.length; j++) {
String key = tableName + "[" + i + "]";
if (!primitiveTable) {
final String columnName = items[j].substring(tableNamePrefix.length(), items[j].length());
key = key + columnName;
}
String value = lineValues.get(items[j]);
if (!org.talend.core.model.utils.ContextParameterUtils.isDynamic(value)) {
value = org.talend.core.model.utils.TalendTextUtils.removeQuotes(value);
value = org.talend.core.model.utils.TalendTextUtils.addQuotes(value);
}
if(value==null || "null".equals(value.trim())) {
value = "(Object)null";
}
%>
component_parameters.put("<%=key%>",String.valueOf(<%=value%>));
<%
}
}
} else if(p.getFieldType() == EParameterFieldType.SCHEMA_TYPE) {
final String parameterName = p.getName();
IConnection connection = null;
final List<? extends IConnection> connections = NodeUtil.getOutgoingConnections(node, p.getContext());
if(connections != null && !connections.isEmpty()) {
connection = connections.get(0);
}
if(connection != null) {
IMetadataTable metaTable = connection.getMetadataTable();
List<IMetadataColumn> columns = metaTable.getListColumns();
for(int i = 0; i < columns.size(); i++) {
IMetadataColumn column = columns.get(i);
%>
component_parameters.put("<%=parameterName%>[<%=i%>]", "<%=column.getLabel()%>");
<%
}
}
} else if (p.getFieldType() == EParameterFieldType.TACOKIT_INPUT_SCHEMA) {
final String parameterName = p.getName();
IConnection connection = null;
final List<? extends IConnection> connections = NodeUtil.getIncomingConnections(node, p.getContext());
if(connections != null && !connections.isEmpty()) {
connection = connections.get(0);
}
if(connection != null) {
IMetadataTable metaTable = connection.getMetadataTable();
List<IMetadataColumn> columns = metaTable.getListColumns();
for(int i = 0; i < columns.size(); i++) {
IMetadataColumn column = columns.get(i);
%>
component_parameters.put("<%=parameterName%>[<%=i%>]", "<%=column.getLabel()%>");
<%
}
}
} else {
final String key;
if(!p.getName().contains("$")){
key = p.getName();
}else{
final StringBuilder keyBuilder = new StringBuilder();
for (String part : p.getName().split("\\.")) {
if (keyBuilder.length() != 0) {
keyBuilder.append(".");
}
if (part.contains("$") && !part.startsWith("$")) {
keyBuilder.append(part.replace("$", "."));
} else {
keyBuilder.append(part);
}
}
key = keyBuilder.toString();
}
String value = null;
if(p.getFieldType() == EParameterFieldType.PASSWORD) {
continue;
} else {
value = ElementParameterParser.getStringElementParameterValue(p);
if (!org.talend.core.model.utils.ContextParameterUtils.isDynamic(value)) {
value = org.talend.core.model.utils.TalendTextUtils.removeQuotes(value);
value = org.talend.core.model.utils.TalendTextUtils.addQuotes(value);
}
}
if (value != null) {
if(key.endsWith("$maxBatchSize")){
%>
<%
} else if(p.getFieldType() == EParameterFieldType.CLOSED_LIST) {
String valueTemp = org.talend.core.model.utils.TalendTextUtils.removeQuotes(value);
if ("".equals(valueTemp)) {
String[] listItemsDisplayCodeValue = p.getListItemsDisplayCodeName();
if(listItemsDisplayCodeValue != null && listItemsDisplayCodeValue.length > 0){
valueTemp = listItemsDisplayCodeValue[0];
value = org.talend.core.model.utils.TalendTextUtils.addQuotes(valueTemp);
}
}
}
if(value==null || "null".equals(value.trim())) {
value = "(Object)null";
}
%>
component_parameters.put("<%=key%>", String.valueOf(<%=value%>));
<%
} // else do not put value in configuration
}
%>

View File

@@ -68,6 +68,14 @@
id="org.talend.designer.components.model.UserComponentsProvider">
</ComponentsProvider>
</extension>
<extension
point="org.talend.core.components_provider">
<ComponentsProvider
class="org.talend.designer.codegen.components.model.SharedStudioUserComponentProvider"
folderName="user"
id="org.talend.designer.codegen.components.model.SharedStudioUserComponentProvider">
</ComponentsProvider>
</extension>
<extension
point="org.eclipse.core.runtime.preferences">
<initializer

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.codegen</artifactId>

View File

@@ -66,6 +66,7 @@ import org.talend.designer.codegen.model.CodeGeneratorEmittersPoolFactory;
import org.talend.designer.codegen.model.CodeGeneratorInternalTemplatesFactoryProvider;
import org.talend.designer.codegen.proxy.JetProxy;
import org.talend.designer.core.generic.model.Component;
import org.talend.designer.runprocess.ProcessorUtilities;
/**
* CodeGenerator.
@@ -353,8 +354,12 @@ public class CodeGenerator implements ICodeGenerator {
.getDefault().getService(IResourcesDependenciesService.class);
String resourcePathForContext = null;
if (process instanceof IProcess2) {
resourcePathForContext = resourceService.getResourcePathForContext(process,
contextPar.getValue());
String value = contextPar.getValue();
// for runtime
if (!ProcessorUtilities.isExportConfig()) {
contextPar.setType(JavaTypesManager.FILE.getId());
}
resourcePathForContext = resourceService.getResourcePathForContext(process, value);
} else {
// for PreviewFileInputContentDataProcess run
resourcePathForContext = resourceService.getResourceItemFilePath(contextPar.getValue());

View File

@@ -69,6 +69,15 @@ public class JavaRoutineSynchronizer extends AbstractRoutineSynchronizer {
syncRoutineItems(getRoutines(true), true);
}
@Override
public void syncAllInnerCodes() throws SystemException {
syncInnerCodeItems(false);
}
@Override
public void syncAllInnerCodesForLogOn() throws SystemException {
syncInnerCodeItems(true);
}
private void syncRoutineItems(Collection<RoutineItem> routineObjects, boolean forceUpdate) throws SystemException {
for (RoutineItem routineItem : routineObjects) {

View File

@@ -26,11 +26,9 @@ import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.ResourceBundle;
import java.util.Optional;
import java.util.Set;
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.Lock;
@@ -70,12 +68,12 @@ import org.talend.core.model.components.ComponentCategory;
import org.talend.core.model.components.ComponentManager;
import org.talend.core.model.components.ComponentProviderInfo;
import org.talend.core.model.components.ComponentUtilities;
import org.talend.core.model.components.EComponentType;
import org.talend.core.model.components.IComponent;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.model.components.IComponentsHandler;
import org.talend.core.model.components.filters.ComponentsFactoryProviderManager;
import org.talend.core.model.components.filters.IComponentFactoryFilter;
import org.talend.core.runtime.util.ComponentsLocationProvider;
import org.talend.core.ui.IJobletProviderService;
import org.talend.core.ui.ISparkJobletProviderService;
import org.talend.core.ui.ISparkStreamingJobletProviderService;
@@ -84,8 +82,6 @@ import org.talend.core.ui.images.CoreImageProvider;
import org.talend.core.utils.TalendCacheUtils;
import org.talend.designer.codegen.CodeGeneratorActivator;
import org.talend.designer.codegen.i18n.Messages;
import org.talend.designer.core.ITisLocalProviderService;
import org.talend.designer.core.ITisLocalProviderService.ResClassLoader;
import org.talend.designer.core.model.components.ComponentBundleToPath;
import org.talend.designer.core.model.components.ComponentFilesNaming;
import org.talend.designer.core.model.components.EmfComponent;
@@ -165,7 +161,11 @@ public class ComponentsFactory implements IComponentsFactory {
throw new RuntimeException(e);
}
isInitialising.set(true);
removeOldComponentsUserFolder(); // not used anymore
try {
removeOldComponentsUserFolder();
} catch (IOException ex) {
ExceptionHandler.process(ex);
} // not used anymore
long startTime = System.currentTimeMillis();
// TimeMeasure.display = true;
@@ -388,10 +388,12 @@ public class ComponentsFactory implements IComponentsFactory {
ComponentManager.saveResource();
}
private void removeOldComponentsUserFolder() {
private void removeOldComponentsUserFolder() throws IOException {
String userPath = IComponentsFactory.COMPONENTS_INNER_FOLDER + File.separatorChar
+ ComponentUtilities.getExtFolder(OLD_COMPONENTS_USER_INNER_FOLDER);
File componentsLocation = getComponentsLocation(userPath);
ComponentsProviderManager componentsProviderManager = ComponentsProviderManager.getInstance();
AbstractComponentsProvider componentsProvider = componentsProviderManager.loadUserComponentsProvidersFromExtension();
File componentsLocation = getComponentsLocation(componentsProvider, userPath);
if (componentsLocation != null && componentsLocation.exists()) {
FilesUtils.removeFolder(componentsLocation, true);
}
@@ -672,114 +674,38 @@ public class ComponentsFactory implements IComponentsFactory {
*
* @param currentFolder
* @return
* @throws IOException
* @throws BusinessException
*/
private File getComponentsLocation(String folder) {
String componentsPath = IComponentsFactory.COMPONENTS_LOCATION;
IBrandingService breaningService = (IBrandingService) GlobalServiceRegister.getDefault()
.getService(IBrandingService.class);
if (breaningService.isPoweredOnlyCamel()) {
componentsPath = IComponentsFactory.CAMEL_COMPONENTS_LOCATION;
}
Bundle b = Platform.getBundle(componentsPath);
File file = null;
try {
URL url = FileLocator.find(b, new Path(folder), null);
if (url == null) {
return null;
private File getComponentsLocation(AbstractComponentsProvider componentsProvider, String folder) throws IOException {
if (componentsProvider instanceof ComponentsLocationProvider) {
return componentsProvider.getInstallationFolder();
} else {
String componentsPath = IComponentsFactory.COMPONENTS_LOCATION;
IBrandingService breaningService = (IBrandingService) GlobalServiceRegister.getDefault()
.getService(IBrandingService.class);
if (breaningService.isPoweredOnlyCamel()) {
componentsPath = IComponentsFactory.CAMEL_COMPONENTS_LOCATION;
}
URL fileUrl = FileLocator.toFileURL(url);
file = new File(fileUrl.getPath());
} catch (Exception e) {
// e.printStackTrace();
ExceptionHandler.process(e);
}
Bundle b = Platform.getBundle(componentsPath);
return file;
}
private File getComponentsLocation(String folder, AbstractComponentsProvider provider) {
File file = null;
try {
if (provider != null) {
file = provider.getInstallationFolder();
} else {
String componentsPath = IComponentsFactory.COMPONENTS_LOCATION;
Bundle b = Platform.getBundle(componentsPath);
IBrandingService breaningService = (IBrandingService) GlobalServiceRegister.getDefault()
.getService(IBrandingService.class);
if (breaningService.isPoweredOnlyCamel()) {
componentsPath = IComponentsFactory.CAMEL_COMPONENTS_LOCATION;
}
File file = null;
try {
URL url = FileLocator.find(b, new Path(folder), null);
if (url == null) {
return null;
}
URL fileUrl = FileLocator.toFileURL(url);
file = new File(fileUrl.getPath());
} catch (Exception e) {
// e.printStackTrace();
ExceptionHandler.process(e);
}
} catch (Exception e) {
ExceptionHandler.process(e);
}
return file;
}
private ResourceBundle getComponentResourceBundle(IComponent currentComp, String source, String cachedPathSource,
AbstractComponentsProvider provider) {
try {
AbstractComponentsProvider currentProvider = provider;
if (currentProvider == null) {
ComponentsProviderManager componentsProviderManager = ComponentsProviderManager.getInstance();
Collection<AbstractComponentsProvider> providers = componentsProviderManager.getProviders();
for (AbstractComponentsProvider curProvider : providers) {
String path = new Path(curProvider.getInstallationFolder().toString()).toPortableString();
if (source.startsWith(path)) {
// fix for TDI-19889 and TDI-20507 to get the correct component provider
if (cachedPathSource != null) {
if (path.contains(cachedPathSource)) {
currentProvider = curProvider;
break;
}
} else {
currentProvider = curProvider;
break;
}
}
}
}
String installPath = currentProvider.getInstallationFolder().toString();
String label = ComponentFilesNaming.getInstance().getBundleName(currentComp.getName(),
installPath.substring(installPath.lastIndexOf(IComponentsFactory.COMPONENTS_INNER_FOLDER)));
if (currentProvider.isUseLocalProvider()) {
// if the component use local provider as storage (for user / ecosystem components)
// then get the bundle resource from the current main component provider.
// note: code here to review later, service like this shouldn't be used...
ResourceBundle bundle = null;
IBrandingService brandingService = (IBrandingService) GlobalServiceRegister.getDefault()
.getService(IBrandingService.class);
if (brandingService.isPoweredOnlyCamel()) {
bundle = currentProvider.getResourceBundle(label);
} else {
ITisLocalProviderService service = (ITisLocalProviderService) GlobalServiceRegister.getDefault()
.getService(ITisLocalProviderService.class);
bundle = service.getResourceBundle(label);
}
return bundle;
} else {
ResourceBundle bundle = ResourceBundle.getBundle(label, Locale.getDefault(),
new ResClassLoader(currentProvider.getClass().getClassLoader()));
return bundle;
}
} catch (IOException e) {
ExceptionHandler.process(e);
}
return null;
return file;
}
}
private String getCodeLanguageSuffix() {
@@ -838,23 +764,21 @@ public class ComponentsFactory implements IComponentsFactory {
init(false);
}
for (IComponent comp : componentList) {
if (comp.getComponentType() != EComponentType.JOBLET) {
continue;
}
String comName = comp.getName();
if (comp != null && paletteType.equals(comp.getPaletteType())) {
if (comName.equals(name)) {
return comp;
} else if (new JobletUtil().matchExpression(comName)) {
String[] names = comName.split(":"); //$NON-NLS-1$
comName = names[1];
if (comName.equals(name)) {
return comp;
}
// check if reference joblet component presents
JobletUtil jobletUtils = new JobletUtil();
Optional<IComponent> result = jobletUtils.findComponentByName(componentList, name, paletteType);
if (!result.isPresent()) {
// check if any name matching joblet component presents
if (jobletUtils.matchExpression(name)) {
name = StringUtils.substringAfterLast(name, ":"); //$NON-NLS-1$
if (StringUtils.isNotBlank(name)) {
result = jobletUtils.findComponentByName(componentList, name, paletteType);
}
}
}
if (result.isPresent()) {
return result.get();
}
return null;
}
@@ -1085,5 +1009,13 @@ public class ComponentsFactory implements IComponentsFactory {
public void setComponentsHandler(IComponentsHandler componentsHandler) {
this.componentsHandler = componentsHandler;
}
public String getCustomComponentBundlePath() {
ComponentsProviderManager componentsProviderManager = ComponentsProviderManager.getInstance();
AbstractComponentsProvider componentsProvider = componentsProviderManager.loadUserComponentsProvidersFromExtension();
String bundle = componentsProvider.getComponentsBundle();
return ComponentBundleToPath.getPathFromBundle(bundle);
}
}

View File

@@ -23,6 +23,7 @@ import org.eclipse.core.runtime.IExtensionRegistry;
import org.eclipse.core.runtime.Platform;
import org.talend.core.GlobalServiceRegister;
import org.talend.core.model.components.AbstractComponentsProvider;
import org.talend.core.runtime.util.SharedStudioInfoProvider;
import org.talend.core.ui.branding.IBrandingService;
import org.talend.designer.codegen.i18n.Messages;
@@ -69,6 +70,9 @@ public final class ComponentsProviderManager {
try {
AbstractComponentsProvider componentsProvider = (AbstractComponentsProvider) configurationElement
.createExecutableExtension("class"); //$NON-NLS-1$
if (componentsProvider instanceof SharedStudioInfoProvider && !((SharedStudioInfoProvider)componentsProvider).isSupportCurrentMode()) {
continue;
}
componentsProvider.setId(id);
componentsProvider.setFolderName(folderName);
componentsProvider.setContributer(contributerName);
@@ -81,15 +85,15 @@ public final class ComponentsProviderManager {
}
}
public AbstractComponentsProvider loadUserComponentsProvidersFromExtension() {
if (providers == null) {
loadComponentsProvidersFromExtension();
}
for (AbstractComponentsProvider provider : providers) {
if ("org.talend.designer.components.model.UserComponentsProvider".equals(provider.getId())) {
return provider;
}
}
return null;
}
public AbstractComponentsProvider loadUserComponentsProvidersFromExtension() {
if (providers == null) {
loadComponentsProvidersFromExtension();
}
for (AbstractComponentsProvider provider : providers) {
if (provider instanceof UserComponentsProvider) {
return provider;
}
}
return null;
}
}

View File

@@ -0,0 +1,61 @@
package org.talend.designer.codegen.components.model;
//============================================================================
//
//Copyright (C) 2006-2019 Talend Inc. - www.talend.com
//
//This source code is available under agreement available at
//%InstallDIR%\features\org.talend.rcp.branding.%PRODUCTNAME%\%PRODUCTNAME%license.txt
//
//You should have received a copy of the agreement
//along with this program; if not, write to Talend SA
//9 rue Pages 92150 Suresnes, France
//
//============================================================================
import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ResourceBundle;
import org.eclipse.core.runtime.IPath;
import org.eclipse.core.runtime.Path;
import org.eclipse.core.runtime.Platform;
import org.talend.core.model.components.ComponentUtilities;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.runtime.util.ComponentsLocationProvider;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.designer.core.model.components.ComponentBundleToPath;
public class SharedStudioUserComponentProvider extends UserComponentsProvider implements ComponentsLocationProvider{
@Override
public File getInstallationFolder() throws IOException {
File componentFolder = SharedStudioUtils.getSharedStudioComponentsParentFolder();
IPath path = new Path(IComponentsFactory.COMPONENTS_INNER_FOLDER);
path = path.append(IComponentsFactory.EXTERNAL_COMPONENTS_INNER_FOLDER).append(ComponentUtilities.getExtFolder(getFolderName()));
File installationFolder = new File (componentFolder, path.toOSString());
return installationFolder;
}
public String getComponentsBundle() {
return ComponentBundleToPath.SHARED_STUDIO_CUSTOM_COMPONENT_BUNDLE;
}
public boolean isSupportCurrentMode() {
if (SharedStudioUtils.isSharedStudioMode()) {
return true;
}
return false;
}
@Override
public ResourceBundle getResourceBundle(String label) {
URL configFolderUrl = Platform.getConfigurationLocation().getURL();
URLClassLoader urlLoader = new URLClassLoader(new java.net.URL[]{configFolderUrl});
java.util.ResourceBundle bundle = java.util.ResourceBundle.getBundle( label ,
java.util.Locale.getDefault(), urlLoader );
return bundle;
}
}

View File

@@ -34,13 +34,15 @@ import org.talend.core.model.components.ComponentUtilities;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.model.general.Project;
import org.talend.core.model.repository.ERepositoryObjectType;
import org.talend.core.runtime.util.SharedStudioInfoProvider;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.core.ui.branding.IBrandingService;
import org.talend.designer.codegen.CodeGeneratorActivator;
import org.talend.designer.codegen.components.ui.IComponentPreferenceConstant;
import org.talend.repository.ProjectManager;
/***/
public class UserComponentsProvider extends AbstractCustomComponentsProvider {
public class UserComponentsProvider extends AbstractCustomComponentsProvider implements SharedStudioInfoProvider{
@Override
protected File getExternalComponentsLocation() {
@@ -147,5 +149,11 @@ public class UserComponentsProvider extends AbstractCustomComponentsProvider {
public String getComponentsBundle() {
return IComponentsFactory.COMPONENTS_LOCATION;
}
public boolean isSupportCurrentMode() {
if (SharedStudioUtils.isSharedStudioMode()) {
return false;
}
return true;
}
}

View File

@@ -60,6 +60,7 @@ public enum EInternalTemplate {
// just to have compiled templates
FOOTER_ROUTE("footer_route", "0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$
HEADER_ROUTE("header_route", "0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$
MICROSERVICE_APP_SECURITY("microservice_app_security","0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$
CAMEL_RUNIF("camel_run_if", "0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$
CAMEL_SPECIALLINKS("camel_speciallinks", "0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$
MICROSERVICE_APP("microservice_app", "0.0.1"), //$NON-NLS-1$ //$NON-NLS-2$

View File

@@ -18,6 +18,7 @@ import java.util.Map;
import org.eclipse.core.runtime.Platform;
import org.talend.commons.exception.ExceptionHandler;
import org.talend.commons.utils.StringUtils;
import org.talend.designer.core.model.components.ComponentBundleToPath;
/**
* Jet container for a particular component.
@@ -213,8 +214,17 @@ public class JetBean {
if (pluginIdToBundle.containsKey(pluginId)) {
base = pluginIdToBundle.get(pluginId);
} else {
base = Platform.getBundle(pluginId).getEntry("/").toString(); //$NON-NLS-1$
pluginIdToBundle.put(pluginId, base);
if (ComponentBundleToPath.SHARED_STUDIO_CUSTOM_COMPONENT_BUNDLE.equals(pluginId)) {
base = ComponentBundleToPath.getPathFromBundle(pluginId);
if (!base.endsWith("/")) {
base = base + "/";
}
pluginIdToBundle.put(pluginId, base);
} else {
base = Platform.getBundle(pluginId).getEntry("/").toString(); //$NON-NLS-1$
pluginIdToBundle.put(pluginId, base);
}
}
String result = base + relativeUri;
return result;

View File

@@ -136,13 +136,11 @@ public class TalendJETCompiler extends JETCompiler {
// get the plugin name from fileURI
String refPluginName = matcher.group(1);
// retrieve the plugin URI by pluginName.
Bundle refBundle = Platform.getBundle(refPluginName);
if (refBundle != null) {
String realURI = TemplateUtil.getPlatformUrlOfBundle(refPluginName);
String realURI = TemplateUtil.getPlatformUrlOfBundle(refPluginName);
if (realURI != null) {
// replace the old fileURI to new one by pluginURI
String newFileURI = fileURI.replaceFirst(PLUGIN_VAR_PATTERN.pattern(), realURI);
return newFileURI;
}
}
}

View File

@@ -14,6 +14,7 @@ package org.talend.designer.codegen.config;
import org.eclipse.core.runtime.Platform;
import org.osgi.framework.Bundle;
import org.talend.designer.core.model.components.ComponentBundleToPath;
/**
* CodeGenerator Templates Ressources Utils.
@@ -161,10 +162,25 @@ public class TemplateUtil {
* @return
*/
public static String getPlatformUrlOfBundle(String bundleName) {
Bundle bundle = Platform.getBundle(bundleName);
if (bundle == null) {
return null;
}
return "platform:/plugin/" + bundle.getSymbolicName() + "_" + bundle.getVersion().toString() + "/";
if (ComponentBundleToPath.SHARED_STUDIO_CUSTOM_COMPONENT_BUNDLE.equals(bundleName)) {
String basePath = ComponentBundleToPath.getPathFromBundle(bundleName);
if (!basePath.endsWith("/")) {
basePath = basePath + "/";
}
return basePath;
} else {
Bundle bundle = Platform.getBundle(bundleName);
if (bundle == null) {
return null;
}
StringBuilder sb = new StringBuilder();
sb.append("platform:/plugin/");
sb.append(bundle.getSymbolicName());
sb.append("_");
sb.append(bundle.getVersion().toString());
sb.append("/");
return sb.toString();
}
}
}

View File

@@ -47,6 +47,7 @@ import org.talend.core.ui.component.ComponentsFactoryProvider;
import org.talend.designer.codegen.CodeGeneratorActivator;
import org.talend.designer.codegen.config.TemplateUtil;
import org.talend.designer.codegen.i18n.Messages;
import org.talend.designer.core.model.components.ComponentBundleToPath;
/**
* DOC xtan
@@ -256,10 +257,9 @@ public final class JetSkeletonManager {
};
for (TemplateUtil template : CodeGeneratorInternalTemplatesFactoryProvider.getInstance().getTemplates()) {
Bundle b = Platform.getBundle(template.getJetPluginRepository());
URL resourcesUrl = null;
try {
resourcesUrl = FileLocator.toFileURL(FileLocator.find(b, new Path(template.getTemplateRelativeUri()), null));
resourcesUrl = FileLocator.toFileURL(ComponentBundleToPath.findComponentsBundleURL(template.getJetPluginRepository(), new Path(template.getTemplateRelativeUri()), null));
} catch (IOException e) {
ExceptionHandler.process(e);
}

View File

@@ -8,6 +8,7 @@ Require-Bundle: org.eclipse.core.runtime,
org.eclipse.ui,
org.apache.log4j,
org.apache.commons.collections,
org.apache.commons.discovery,
org.apache.commons.logging,
org.apache.commons.beanutils,
org.apache.commons.io,
@@ -25,7 +26,6 @@ Require-Bundle: org.eclipse.core.runtime,
org.talend.repository,
org.talend.core.repository,
org.talend.updates.runtime,
org.apache.axis,
org.eclipse.ui.intro,
org.eclipse.ui.forms,
org.eclipse.jface.text

View File

@@ -9,6 +9,14 @@
id="org.talend.designer.components.exchange.ExchangeComponentsProvider">
</ComponentsProvider>
</extension>
<extension
point="org.talend.core.components_provider">
<ComponentsProvider
class="org.talend.designer.components.exchange.SharedStudioExchangeComponentsProvider"
folderName="exchange"
id="org.talend.designer.components.exchange.SharedStudioExchangeComponentsProvider">
</ComponentsProvider>
</extension>
<extension
point="org.talend.core.runtime.service">
<Service

View File

@@ -4,7 +4,7 @@
<parent>
<groupId>org.talend.studio</groupId>
<artifactId>tdi-studio-se</artifactId>
<version>7.3.1-SNAPSHOT</version>
<version>7.3.1-PATCH</version>
<relativePath>../../../</relativePath>
</parent>
<artifactId>org.talend.designer.components.exchange</artifactId>

View File

@@ -28,13 +28,15 @@ import org.talend.core.GlobalServiceRegister;
import org.talend.core.model.components.AbstractComponentsProvider;
import org.talend.core.model.components.ComponentUtilities;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.runtime.util.SharedStudioInfoProvider;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.core.ui.branding.IBrandingService;
import org.talend.designer.components.exchange.util.ExchangeUtils;
/**
* DOC hcyi class global comment. Detailled comment
*/
public class ExchangeComponentsProvider extends AbstractComponentsProvider {
public class ExchangeComponentsProvider extends AbstractComponentsProvider implements SharedStudioInfoProvider{
/**
* ExchangeComponentsProvider constructor.
@@ -184,4 +186,10 @@ public class ExchangeComponentsProvider extends AbstractComponentsProvider {
return IComponentsFactory.COMPONENTS_LOCATION;
}
public boolean isSupportCurrentMode() {
if (SharedStudioUtils.isSharedStudioMode()) {
return false;
}
return true;
}
}

View File

@@ -0,0 +1,59 @@
package org.talend.designer.components.exchange;
//============================================================================
//
//Copyright (C) 2006-2019 Talend Inc. - www.talend.com
//
//This source code is available under agreement available at
//%InstallDIR%\features\org.talend.rcp.branding.%PRODUCTNAME%\%PRODUCTNAME%license.txt
//
//You should have received a copy of the agreement
//along with this program; if not, write to Talend SA
//9 rue Pages 92150 Suresnes, France
//
//============================================================================
import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ResourceBundle;
import org.eclipse.core.runtime.IPath;
import org.eclipse.core.runtime.Path;
import org.eclipse.core.runtime.Platform;
import org.talend.core.model.components.ComponentUtilities;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.runtime.util.ComponentsLocationProvider;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.designer.core.model.components.ComponentBundleToPath;
public class SharedStudioExchangeComponentsProvider extends ExchangeComponentsProvider implements ComponentsLocationProvider{
@Override
public File getInstallationFolder() throws IOException {
File componentFolder = SharedStudioUtils.getSharedStudioComponentsParentFolder();
IPath path = new Path(IComponentsFactory.COMPONENTS_INNER_FOLDER);
path = path.append(IComponentsFactory.EXTERNAL_COMPONENTS_INNER_FOLDER).append(ComponentUtilities.getExtFolder(getFolderName()));
File installationFolder = new File (componentFolder, path.toOSString());
return installationFolder;
}
public String getComponentsBundle() {
return ComponentBundleToPath.SHARED_STUDIO_CUSTOM_COMPONENT_BUNDLE;
}
public boolean isSupportCurrentMode() {
if (SharedStudioUtils.isSharedStudioMode()) {
return true;
}
return false;
}
@Override
public ResourceBundle getResourceBundle(String label) {
URL configFolderUrl = Platform.getConfigurationLocation().getURL();
URLClassLoader urlLoader = new URLClassLoader(new java.net.URL[]{configFolderUrl});
java.util.ResourceBundle bundle = java.util.ResourceBundle.getBundle( label ,
java.util.Locale.getDefault(), urlLoader );
return bundle;
}
}

View File

@@ -1,66 +0,0 @@
// ============================================================================
//
// Copyright (C) 2006-2019 Talend Inc. - www.talend.com
//
// This source code is available under agreement available at
// %InstallDIR%\features\org.talend.rcp.branding.%PRODUCTNAME%\%PRODUCTNAME%license.txt
//
// You should have received a copy of the agreement
// along with this program; if not, write to Talend SA
// 9 rue Pages 92150 Suresnes, France
//
// ============================================================================
package org.talend.designer.components.exchange.proxy;
import org.apache.commons.lang.StringUtils;
/**
*
* DOC hcyi class global comment. Detailled comment
*/
public class DefaultHTTPSTransportClientProperties extends DefaultHTTPTransportClientProperties {
/**
* @see org.apache.axis.components.net.TransportClientProperties#getProxyHost()
*/
@Override
public String getProxyHost() {
return StringUtils.trimToEmpty(System.getProperty("https.proxyHost")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getNonProxyHosts()
*/
@Override
public String getNonProxyHosts() {
return StringUtils.trimToEmpty(System.getProperty("https.nonProxyHosts")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getPort()
*/
@Override
public String getProxyPort() {
return StringUtils.trimToEmpty(System.getProperty("https.proxyPort")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getUser()
*/
@Override
public String getProxyUser() {
return StringUtils.trimToEmpty(System.getProperty("https.proxyUser")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getPassword()
*/
@Override
public String getProxyPassword() {
return StringUtils.trimToEmpty(System.getProperty("https.proxyPassword")); //$NON-NLS-1$
}
}

View File

@@ -1,58 +0,0 @@
// ============================================================================
//
// Copyright (C) 2006-2019 Talend Inc. - www.talend.com
//
// This source code is available under agreement available at
// %InstallDIR%\features\org.talend.rcp.branding.%PRODUCTNAME%\%PRODUCTNAME%license.txt
//
// You should have received a copy of the agreement
// along with this program; if not, write to Talend SA
// 9 rue Pages 92150 Suresnes, France
//
// ============================================================================
package org.talend.designer.components.exchange.proxy;
import org.apache.axis.components.net.TransportClientProperties;
import org.apache.commons.lang.StringUtils;
/**
*
* DOC hcyi class global comment. Detailled comment
*/
public class DefaultHTTPTransportClientProperties implements TransportClientProperties {
/**
* @see org.apache.axis.components.net.TransportClientProperties#getProxyHost()
*/
public String getProxyHost() {
return StringUtils.trimToEmpty(System.getProperty("http.proxyHost")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getNonProxyHosts()
*/
public String getNonProxyHosts() {
return StringUtils.trimToEmpty(System.getProperty("http.nonProxyHosts")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getPort()
*/
public String getProxyPort() {
return StringUtils.trimToEmpty(System.getProperty("http.proxyPort")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getProxyUser()
*/
public String getProxyUser() {
return StringUtils.trimToEmpty(System.getProperty("http.proxyUser")); //$NON-NLS-1$
}
/**
* @see org.apache.axis.components.net.TransportClientProperties#getProxyPassword()
*/
public String getProxyPassword() {
return StringUtils.trimToEmpty(System.getProperty("http.proxyPassword")); //$NON-NLS-1$
}
}

View File

@@ -52,6 +52,7 @@ import org.talend.core.download.IDownloadHelper;
import org.talend.core.model.components.ComponentManager;
import org.talend.core.model.components.IComponent;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.core.ui.component.ComponentPaletteUtilities;
import org.talend.core.ui.component.ComponentsFactoryProvider;
import org.talend.designer.codegen.ICodeGeneratorService;
@@ -312,51 +313,54 @@ public class DownloadComponenentsAction extends Action implements IIntroAction {
protected void afterDownload(IProgressMonitor monitor, ComponentExtension extension, File localZipFile) throws Exception {
if (UpdatesHelper.isComponentUpdateSite(localZipFile)) {
final File workFolder = org.talend.utils.files.FileUtils.createTmpFolder("downloadedComponents", ""); //$NON-NLS-1$ //$NON-NLS-2$
if (!SharedStudioUtils.isSharedStudioMode()) {
final File workFolder = org.talend.utils.files.FileUtils.createTmpFolder("downloadedComponents", ""); //$NON-NLS-1$ //$NON-NLS-2$
try {
FilesUtils.copyFile(localZipFile, new File(workFolder, localZipFile.getName()));
try {
FilesUtils.copyFile(localZipFile, new File(workFolder, localZipFile.getName()));
ComponentsInstallComponent component = LocalComponentInstallHelper.getComponent();
if (component != null) {
try {
component.setComponentFolder(workFolder);
if (component.install()) {
ComponentsInstallComponent component = LocalComponentInstallHelper.getComponent();
if (component != null) {
try {
component.setComponentFolder(workFolder);
if (component.install()) {
if (component.needRelaunch()) {
askReboot();
} else {
MessageDialog.openInformation(DisplayUtils.getDefaultShell(),
Messages.getString("DownloadComponenentsAction.installComponentsTitle"),
component.getInstalledMessages());
if (component.needRelaunch()) {
askReboot();
} else {
MessageDialog.openInformation(DisplayUtils.getDefaultShell(),
Messages.getString("DownloadComponenentsAction.installComponentsTitle"),
component.getInstalledMessages());
}
} else {// install failure
MessageDialog.openWarning(DisplayUtils.getDefaultShell(),
Messages.getString("DownloadComponenentsAction_failureTitle"), //$NON-NLS-1$
Messages.getString("DownloadComponenentsAction_failureMessage", extension.getLabel())); //$NON-NLS-1$
}
} else {// install failure
MessageDialog.openWarning(DisplayUtils.getDefaultShell(),
} finally {
// after install, clear the setting for service.
component.setComponentFolder(null);
}
}
} catch (Exception e) {
// Popup dialog to user to waring install failed.
Display.getDefault().syncExec(new Runnable() {
@Override
public void run() {
MessageDialog.openError(DisplayUtils.getDefaultShell(false),
Messages.getString("DownloadComponenentsAction_failureTitle"), //$NON-NLS-1$
Messages.getString("DownloadComponenentsAction_failureMessage", extension.getLabel())); //$NON-NLS-1$
}
} finally {
// after install, clear the setting for service.
component.setComponentFolder(null);
}
});
throw e;
} finally {
FilesUtils.deleteFolder(workFolder, true);
}
} catch (Exception e) {
// Popup dialog to user to waring install failed.
Display.getDefault().syncExec(new Runnable() {
@Override
public void run() {
MessageDialog.openError(DisplayUtils.getDefaultShell(false),
Messages.getString("DownloadComponenentsAction_failureTitle"), //$NON-NLS-1$
Messages.getString("DownloadComponenentsAction_failureMessage", extension.getLabel())); //$NON-NLS-1$
}
});
throw e;
} finally {
FilesUtils.deleteFolder(workFolder, true);
}
monitor.done();
ExchangeManager.getInstance().saveDownloadedExtensionsToFile(extension);
monitor.done();
ExchangeManager.getInstance().saveDownloadedExtensionsToFile(extension);
}
} else {
File installedLocation = ComponentInstaller.unzip(localZipFile.getAbsolutePath(), getComponentsFolder()
.getAbsolutePath());

View File

@@ -37,6 +37,7 @@ import org.eclipse.swt.widgets.Shell;
import org.eclipse.ui.PlatformUI;
import org.talend.commons.ui.runtime.exception.ExceptionHandler;
import org.talend.core.download.DownloadHelper;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.designer.components.exchange.i18n.Messages;
import org.talend.designer.components.exchange.model.Category;
import org.talend.designer.components.exchange.model.VersionRevision;
@@ -105,7 +106,7 @@ public class ImportExchangeDialog extends Dialog {
@Override
protected void okPressed() {
IPath tempPath = new Path(System.getProperty("user.dir")).append("temp"); //$NON-NLS-1$ //$NON-NLS-2$
IPath tempPath = SharedStudioUtils.getTempFolderPath();
File pathFile = tempPath.toFile();
if (downloadproperty.getFileName() == null || downloadproperty.getFileName() == null) {
MessageBox box = new MessageBox(Display.getCurrent().getActiveShell(), SWT.ICON_WARNING | SWT.OK);

View File

@@ -25,11 +25,9 @@ import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.axis.components.net.TransportClientProperties;
import org.apache.axis.components.net.TransportClientPropertiesFactory;
import org.apache.commons.beanutils.BeanUtils;
import org.apache.commons.collections.map.MultiValueMap;
import org.apache.commons.discovery.tools.ManagedProperties;
import org.apache.commons.httpclient.HostConfiguration;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.NameValuePair;
@@ -55,6 +53,7 @@ import org.talend.core.language.ECodeLanguage;
import org.talend.core.language.LanguageManager;
import org.talend.core.model.components.IComponentsFactory;
import org.talend.core.model.general.Project;
import org.talend.core.runtime.util.SharedStudioUtils;
import org.talend.core.ui.component.ComponentPaletteUtilities;
import org.talend.core.ui.component.ComponentsFactoryProvider;
import org.talend.designer.components.exchange.ExchangePlugin;
@@ -150,14 +149,17 @@ public class ExchangeUtils {
public static String sendGetRequest(String urlAddress) throws Exception {
HttpClient httpclient = new HttpClient();
GetMethod getMethod = new GetMethod(urlAddress);
TransportClientProperties tcp = TransportClientPropertiesFactory.create("http");
if (tcp.getProxyHost().length() != 0) {
String proxyUser = ManagedProperties.getProperty("http.proxyUser");
String proxyPassword = ManagedProperties.getProperty("http.proxyPassword");
String proxyHost = ManagedProperties.getProperty("http.proxyHost");
proxyHost = proxyHost != null ? proxyHost : "";
String proxyPort = ManagedProperties.getProperty("http.proxyPort");
if (proxyHost.length() != 0) {
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(
tcp.getProxyUser() != null ? tcp.getProxyUser() : "",
tcp.getProxyPassword() != null ? tcp.getProxyUser() : "");
proxyUser != null ? proxyUser : "", proxyPassword != null ? proxyPassword : "");
httpclient.getState().setProxyCredentials(AuthScope.ANY, creds);
HostConfiguration hcf = new HostConfiguration();
hcf.setProxy(tcp.getProxyHost(), Integer.parseInt(tcp.getProxyPort()));
hcf.setProxy(proxyHost, Integer.parseInt(proxyPort));
httpclient.executeMethod(hcf, getMethod);
} else {
httpclient.executeMethod(getMethod);
@@ -205,14 +207,19 @@ public class ExchangeUtils {
* @return
*/
public static File getComponentFolder(String componentfolder) {
URL url = FileLocator.find(ExchangePlugin.getDefault().getBundle(), new Path(componentfolder), null);
try {
URL fileUrl = FileLocator.toFileURL(url);
return new File(fileUrl.getPath());
} catch (Exception e) {
ExceptionHandler.process(e);
}
return null;
if (SharedStudioUtils.isSharedStudioMode()) {
File componentFolder = SharedStudioUtils.getSharedStudioComponentsExtFolder();
return new File (componentFolder, componentfolder);
} else {
URL url = FileLocator.find(ExchangePlugin.getDefault().getBundle(), new Path(componentfolder), null);
try {
URL fileUrl = FileLocator.toFileURL(url);
return new File(fileUrl.getPath());
} catch (Exception e) {
ExceptionHandler.process(e);
}
return null;
}
}
/**

View File

@@ -11,7 +11,7 @@
<!-- modification 2: compile classpath -->
<path id="compile.classpath">
<pathelement location="../../../../../../tcommon-studio-se/main/plugins/org.talend.libraries.dom4j-jaxen/lib/dom4j-1.6.1.jar" />
<pathelement location="../../../../../../tcommon-studio-se/main/plugins/org.talend.libraries.dom4j-jaxen/lib/dom4j-2.1.3.jar" />
<pathelement location="../../../../../../tcommon-studio-se/main/plugins/org.talend.libraries.apache.common/lib/commons-lang-2.6.jar" />
</path>

View File

@@ -2,7 +2,6 @@
<project name="org.talend.designer.components.libs" default="buildall" basedir=".">
<target name="buildall">
<ant antfile="talend_file_enhanced_20070724/build.xml" target="process" inheritall="no" />
<ant antfile="sugarCRMManagement/build.xml" target="process" inheritall="no" />
<ant antfile="TalendSAX/build.xml" target="process" inheritall="no" />
</target>

View File

@@ -1,8 +1,8 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.talend.libraries</groupId>
<artifactId>checkArchive-1.1-20190917</artifactId>
<version>6.0.0</version>
<groupId>org.talend.components</groupId>
<artifactId>checkArchive</artifactId>
<version>1.2-20210901</version>
<name>checkArchive</name>
<description>Dependence for tFileArchive and tFileUnAchive</description>
<url>http://maven.apache.org</url>
@@ -10,7 +10,7 @@
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<talend.nexus.url>https://artifacts-oss.talend.com</talend.nexus.url>
<java.source.version>1.7</java.source.version>
<java.source.version>1.8</java.source.version>
</properties>
<distributionManagement>
@@ -41,7 +41,7 @@
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
<version>1.19</version>
<version>1.21</version>
</dependency>
</dependencies>
<build>

View File

@@ -3,11 +3,32 @@
<modelVersion>4.0.0</modelVersion>
<groupId>org.talend.components.lib</groupId>
<artifactId>commons-net-ftps-proxy</artifactId>
<version>3.6.1-talend-20190819</version>
<version>3.6.1-talend-20200902</version>
<name>commons-net-talend</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<talend.nexus.url>https://artifacts-oss.talend.com</talend.nexus.url>
<slf4.version>1.7.25</slf4.version>
<lombok.version>1.18.12</lombok.version>
</properties>
<dependencies>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
@@ -15,10 +36,6 @@
</dependency>
</dependencies>
<properties>
<talend.nexus.url>https://artifacts-oss.talend.com</talend.nexus.url>
</properties>
<distributionManagement>
<snapshotRepository>
<id>talend_nexus_deployment</id>

View File

@@ -11,8 +11,10 @@ import javax.net.ssl.SSLSession;
import javax.net.ssl.SSLSessionContext;
import javax.net.ssl.SSLSocket;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.net.ftp.FTPSClient;
@Slf4j
public class SSLSessionReuseFTPSClient extends FTPSClient {
public SSLSessionReuseFTPSClient(boolean isImplicit, SSLContext context) {
@@ -24,6 +26,12 @@ public class SSLSessionReuseFTPSClient extends FTPSClient {
if (socket instanceof SSLSocket) {
final SSLSession session = ((SSLSocket) _socket_).getSession();
final SSLSessionContext context = session.getSessionContext();
if (context == null) {
// TDI-44654 (may be reproduced with Syncplify server)
log.info("SSL Session Context is null. SSL Session was re-initialized.");
return;
}
try {
final Field sessionHostPortCache = context.getClass().getDeclaredField("sessionHostPortCache");
sessionHostPortCache.setAccessible(true);
@@ -32,10 +40,10 @@ public class SSLSessionReuseFTPSClient extends FTPSClient {
putMethod.setAccessible(true);
InetAddress address = socket.getInetAddress();
int port = socket.getPort();
String key = String.format("%s:%s", address.getHostName(), String.valueOf(port)).toLowerCase(Locale.ROOT);
putMethod.invoke(cache, key, session);
key = String.format("%s:%s", address.getHostAddress(), String.valueOf(port)).toLowerCase(Locale.ROOT);
putMethod.invoke(cache, key, session);
} catch (Exception e) {

View File

@@ -2,9 +2,9 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.talend.libraries</groupId>
<groupId>org.talend.components</groupId>
<artifactId>filecopy</artifactId>
<version>2.0.0</version>
<version>2.0.3</version>
<packaging>jar</packaging>
<name>talend-copy</name>
@@ -14,6 +14,7 @@
<talend.nexus.url>https://artifacts-oss.talend.com</talend.nexus.url>
<java.source.version>1.8</java.source.version>
<junit5.version>5.4.2</junit5.version>
<slf4j.version>1.7.28</slf4j.version>
</properties>
<distributionManagement>
@@ -52,7 +53,12 @@
<version>${junit5.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
<build>
<plugins>

View File

@@ -15,13 +15,21 @@ package org.talend;
import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.FileTime;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* DOC Administrator class global comment. Detailled comment
*/
public class FileCopy {
static Logger logger = LoggerFactory.getLogger(Object.class);
/** Private constructor, only static methods */
private FileCopy() {
}
@@ -34,16 +42,57 @@ public class FileCopy {
* @param delSrc : true if delete source.
* @throws IOException : if IO pb.
*/
public static void copyFile(String srcFileName, String desFileName, boolean delSrc) throws IOException {
final File source = new File(srcFileName);
final File destination = new File(desFileName);
public static void copyFile(String srcFileName, String desFileName, boolean delSrc, boolean keepModified)
throws IOException {
final Path source = Paths.get(srcFileName);
final Path destination = Paths.get(desFileName);
FileTime lastModifiedTime = null;
try {
lastModifiedTime = Files.getLastModifiedTime(source);
} catch (IOException e) {
logger.warn(e.getLocalizedMessage());
}
if (delSrc) {
// move : more efficient if in same FS and mustr delete existing file.
Files.move(source.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING);
Files.move(source, destination, StandardCopyOption.REPLACE_EXISTING);
} else {
Files.copy(source.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING);
Files.copy(source, destination, StandardCopyOption.REPLACE_EXISTING);
}
if(keepModified){
try {
Files.setLastModifiedTime(destination,lastModifiedTime);
} catch (IOException e) {
logger.warn(e.getLocalizedMessage());
}
}
}
public static void copyFile(String srcFileName, String desFileName, boolean delSrc ) throws IOException {
copyFile(srcFileName,desFileName,delSrc,true);
}
/**
* Force Copy and Delete files.
*
* @param srcFileName : file name for source file.
* @param desFileName : file name for destination file.
* @throws IOException : if IO pb.
*/
public static void forceCopyAndDelete(String srcFileName, String desFileName, boolean keepModified) throws IOException {
final Path source = Paths.get(srcFileName);
final Path destination = Paths.get(desFileName);
final long lastModifiedTime = new File(srcFileName).lastModified();
Files.copy(source, destination, StandardCopyOption.REPLACE_EXISTING);
Files.delete(source);
if(keepModified){
destination.toFile().setLastModified(lastModifiedTime);
}
}
public static void forceCopyAndDelete(String srcFileName, String desFileName) throws IOException {
forceCopyAndDelete(srcFileName,desFileName,true);
}
}

View File

@@ -100,6 +100,44 @@ class FileCopyTest {
Assertions.assertEquals(referenceSize, copy.length(), "Size error");
}
@Test
void testForceCopyWithDelete() throws Exception {
final URL repCopy = Thread.currentThread().getContextClassLoader().getResource("copy");
File file = this.buildFile("fileToDelete.txt", 10L * 1024L);
file.deleteOnExit();
File copy = new File(repCopy.getPath(), "fileToDelete.txt");
long referenceSize = file.length();
if (!copy.exists()) {
copy.createNewFile();
}
copy.deleteOnExit();
FileCopy.forceCopyAndDelete(file.getPath(), copy.getPath());
Assertions.assertFalse(file.exists(), "file not delete");
Assertions.assertTrue(copy.exists(), "small file : original file deleted");
Assertions.assertEquals(referenceSize, copy.length(), "Size error");
}
@Test
void testLastModifiedTime() throws Exception {
final URL repCopy = Thread.currentThread().getContextClassLoader().getResource("copy");
File file = this.buildFile("fileLMT.txt", 10L * 1024L);
file.deleteOnExit();
long referencceTime = 324723894L;
file.setLastModified(referencceTime);
File copy = new File(repCopy.getPath(), "fileLMTDestination.txt");
if (copy.exists()) {
copy.delete();
}
copy.deleteOnExit();
FileCopy.copyFile(file.getPath(), copy.getPath(), true);
Assertions.assertEquals(referencceTime, copy.lastModified(), "modified time is not idential");
}
/**
* Generate a new file for testing.
*
@@ -125,4 +163,22 @@ class FileCopyTest {
return generatedFile;
}
@Test
void testKeepLastModifiedTime() throws Exception {
final URL repCopy = Thread.currentThread().getContextClassLoader().getResource("copy");
File file = this.buildFile("fileLMT.txt", 10L * 1024L);
file.deleteOnExit();
long referencceTime = 324723894L;
file.setLastModified(referencceTime);
File copy = new File(repCopy.getPath(), "fileLMTDestination.txt");
if (copy.exists()) {
copy.delete();
}
copy.deleteOnExit();
FileCopy.copyFile(file.getPath(), copy.getPath(), true,true);
Assertions.assertEquals(referencceTime, copy.lastModified(), "modified time is not idential");
}
}

View File

@@ -22,7 +22,7 @@
<groupId>org.talend.components</groupId>
<artifactId>google-storage-utils</artifactId>
<version>1.0.0</version>
<version>1.1.0</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<talend.nexus.url>https://artifacts-oss.talend.com</talend.nexus.url>
@@ -46,6 +46,12 @@
<artifactId>jets3t</artifactId>
<version>0.9.0</version>
</dependency>
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>5.6.2</version>
<scope>test</scope>
</dependency>
</dependencies>
<distributionManagement>
<snapshotRepository>
@@ -64,8 +70,8 @@
<artifactId>maven-compiler-plugin</artifactId>
<version>2.3.2</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
<plugin>

View File

@@ -3,10 +3,13 @@ package org.talend.gs.util;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.FileSystems;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.jets3t.service.model.GSObject;
@@ -44,6 +47,50 @@ public class GSObjectUtil {
return objects;
}
public Map<String, File> generateFileMap(File rootFile, String directory, Map<String, String> filter){
Map<String, File> allFiles = generateFileMap(rootFile, directory);
Map<String, File> resultMap = new HashMap<>();
for (Map.Entry<String, String> entry : filter.entrySet()) {
//key is path to file on local machine
//value is new path in GS
File currentFile = new File(entry.getKey());
if (allFiles.containsValue(currentFile)){
resultMap.put(entry.getValue(), currentFile);
}
}
return resultMap;
}
public Map<String, File> generateFileMap(File rootFile, String directory){
if (rootFile == null){
throw new IllegalArgumentException("File can't be null");
}
if (directory == null){
throw new IllegalArgumentException("Directory name can't be null");
}
if (!directory.isEmpty() && !"/".equals(directory.substring(directory.length()-1))){
directory = directory + "/";
}
Map<String,File> fileMap = new HashMap<>();
if (!rootFile.isDirectory()){
fileMap.put(directory + rootFile.getName(), rootFile);
return fileMap;
}
String pathToDirectory = rootFile.getAbsolutePath();
List<File> allFiles = listAllFiles(pathToDirectory);
for (File currentFile : allFiles){
if (!currentFile.isDirectory()){
fileMap.put(combineTwoNames(pathToDirectory, directory, currentFile), currentFile);
}
}
return fileMap;
}
public java.util.Map<String, String> genFileFilterList(java.util.List<java.util.Map<String, String>> list,
String localdir, String remotedir) {
if (remotedir != null && !"".equals(remotedir)) {
@@ -120,4 +167,23 @@ public class GSObjectUtil {
InputStream mimetypesFile = this.getClass().getResourceAsStream("/resource/mime.types");
Mimetypes.getInstance().loadAndReplaceMimetypes(mimetypesFile);
}
private List<File> listAllFiles(String directoryName) {
File directory = new File(directoryName);
List<File> resultList = new ArrayList<>();
File[] fList = directory.listFiles();
resultList.addAll(Arrays.asList(fList));
for (File file : fList) {
if (file.isDirectory()) {
resultList.addAll(listAllFiles(file.getAbsolutePath()));
}
}
return resultList;
}
private String combineTwoNames(String coreDir, String targetDir, File file){
String filePath = file.getAbsolutePath();
String relativePath = targetDir + filePath.replace(coreDir + FileSystems.getDefault().getSeparator(), "");
return relativePath.replaceAll("\\\\","/");
}
}

View File

@@ -0,0 +1,131 @@
package util;
import org.junit.jupiter.api.Test;
import org.talend.gs.util.GSObjectUtil;
import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Map;
import static org.junit.jupiter.api.Assertions.*;
class GSObjectUtilTest {
@Test
void checkSizeOfFilesInFolderTest() {
Path pathToDirectory = Paths.get("src", "test", "resources", "dirWithThreeFiles");
Path pathToFile1 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "1.csv");
Path pathToFile2 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "2.csv");
Path pathToFile3 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "3.csv");
Path pathToFile4 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "inner", "1.csv");
String expectedRelativePathToFile1 = "/1.csv";
String expectedRelativePathToFile2 = "/2.csv";
String expectedRelativePathToFile3 = "/3.csv";
String expectedRelativePathToFile4 = "/inner/1.csv";
File rootFile = new File(pathToDirectory.toUri());
GSObjectUtil gsObjectUtil = new GSObjectUtil();
Map<String, File> files = gsObjectUtil.generateFileMap(rootFile, "/");
assertEquals(4, files.size());
File expectedFile1 = new File(pathToFile1.toUri());
File expectedFile2 = new File(pathToFile2.toUri());
File expectedFile3 = new File(pathToFile3.toUri());
File expectedFile4 = new File(pathToFile4.toUri());
assertTrue(files.containsKey(expectedRelativePathToFile1));
assertTrue(files.containsKey(expectedRelativePathToFile2));
assertTrue(files.containsKey(expectedRelativePathToFile3));
assertTrue(files.containsKey(expectedRelativePathToFile4));
assertEquals(expectedFile1.getAbsoluteFile(), files.get(expectedRelativePathToFile1).getAbsoluteFile());
assertEquals(expectedFile2.getAbsoluteFile(), files.get(expectedRelativePathToFile2).getAbsoluteFile());
assertEquals(expectedFile3.getAbsoluteFile(), files.get(expectedRelativePathToFile3).getAbsoluteFile());
assertEquals(expectedFile4.getAbsoluteFile(), files.get(expectedRelativePathToFile4).getAbsoluteFile());
}
@Test
void checkSizeOfFilesInFolder2Test() {
Path pathToDirectory = Paths.get("src", "test", "resources", "dirWithThreeFiles");
Path pathToFile1 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "1.csv");
Path pathToFile2 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "2.csv");
Path pathToFile3 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "3.csv");
Path pathToFile4 = Paths.get("src", "test", "resources", "dirWithThreeFiles", "inner", "1.csv");
String expectedRelativePathToFile1 = "1.csv";
String expectedRelativePathToFile2 = "2.csv";
String expectedRelativePathToFile3 = "3.csv";
String expectedRelativePathToFile4 = "inner/1.csv";
File rootFile = new File(pathToDirectory.toUri());
GSObjectUtil gsObjectUtil = new GSObjectUtil();
Map<String, File> files = gsObjectUtil.generateFileMap(rootFile, "");
assertEquals(4, files.size());
File expectedFile1 = new File(pathToFile1.toUri());
File expectedFile2 = new File(pathToFile2.toUri());
File expectedFile3 = new File(pathToFile3.toUri());
File expectedFile4 = new File(pathToFile4.toUri());
assertTrue(files.containsKey(expectedRelativePathToFile1));
assertTrue(files.containsKey(expectedRelativePathToFile2));
assertTrue(files.containsKey(expectedRelativePathToFile3));
assertTrue(files.containsKey(expectedRelativePathToFile4));
assertEquals(expectedFile1.getAbsoluteFile(), files.get(expectedRelativePathToFile1).getAbsoluteFile());
assertEquals(expectedFile2.getAbsoluteFile(), files.get(expectedRelativePathToFile2).getAbsoluteFile());
assertEquals(expectedFile3.getAbsoluteFile(), files.get(expectedRelativePathToFile3).getAbsoluteFile());
assertEquals(expectedFile4.getAbsoluteFile(), files.get(expectedRelativePathToFile4).getAbsoluteFile());
}
@Test
void oneFileWithCustomPathTest(){
Path pathToFile = Paths.get("src","test", "resources", "empty.csv");
String pathToFolder = "/folder/folder";
File expectedFile = new File(pathToFile.toUri());
GSObjectUtil gsObjectUtil = new GSObjectUtil();
Map<String, File> files = gsObjectUtil.generateFileMap(expectedFile, pathToFolder);
String expectedRelativePathToFile = "/folder/folder/empty.csv";
assertEquals(1, files.size());
assertTrue(files.containsKey(expectedRelativePathToFile));
assertEquals(expectedFile.getAbsoluteFile(), files.get(expectedRelativePathToFile).getAbsoluteFile());
}
@Test
void oneFileWithOtherCustomPathTest(){
Path pathToFile = Paths.get("src","test", "resources", "empty.csv");
String pathToFolder = "folder/folder";
File expectedFile = new File(pathToFile.toUri());
GSObjectUtil gsObjectUtil = new GSObjectUtil();
Map<String, File> files = gsObjectUtil.generateFileMap(expectedFile, pathToFolder);
String expectedRelativePathToFile = "folder/folder/empty.csv";
assertEquals(1, files.size());
assertTrue(files.containsKey(expectedRelativePathToFile));
assertEquals(expectedFile.getAbsoluteFile(), files.get(expectedRelativePathToFile).getAbsoluteFile());
}
@Test
void folderWithOtherCustomPathTest(){
Path pathToFolder = Paths.get("src", "test", "resources", "dirWithThreeFiles", "inner");
String pathToGSFolder = "folder";
File folder = new File(pathToFolder.toUri());
GSObjectUtil gsObjectUtil = new GSObjectUtil();
Map<String, File> files = gsObjectUtil.generateFileMap(folder, pathToGSFolder);
String expectedRelativePathToFile = "folder/1.csv";
File expectedFile = new File(pathToFolder + "/1.csv");
assertEquals(1, files.size());
assertTrue(files.containsKey(expectedRelativePathToFile));
assertEquals(expectedFile.getAbsoluteFile(), files.get(expectedRelativePathToFile).getAbsoluteFile());
}
}

View File

@@ -170,7 +170,7 @@
<dependency>
<groupId>commons-beanutils</groupId>
<artifactId>commons-beanutils</artifactId>
<version>1.8.0</version>
<version>1.9.4</version>
<scope>compile</scope>
</dependency>
<dependency>
@@ -310,7 +310,7 @@
<link>http://java.sun.com/j2se/1.5.0/docs/api/</link>
<link>http://www.junit.org/junit/javadoc/</link>
<link>
http://jakarta.apache.org/commons/beanutils/commons-beanutils-1.7.0/docs/api/
http://jakarta.apache.org/commons/beanutils/commons-beanutils-1.9.4/docs/api/
</link>
<link>
http://jakarta.apache.org/commons/lang/api-2.2/

Some files were not shown because too many files have changed in this diff Show More