Commit Graph

192 Commits

Author SHA1 Message Date
Henry Heng 5a37227d14
Chore/refractor (#4454)
* markdown files and env examples cleanup

* components update

* update jsonlines description

* server refractor

* update telemetry

* add execute custom node

* add ui refractor

* add username and password authenticate

* correctly retrieve past images in agentflowv2

* disable e2e temporarily

* add existing username and password authenticate

* update migration to default workspace

* update todo

* blob storage migrating

* throw error on agent tool call error

* add missing execution import

* add referral

* chore: add error message when importData is undefined

* migrate api keys to db

* fix: data too long for column executionData

* migrate api keys from json to db at init

* add info on account setup

* update docstore missing fields

---------

Co-authored-by: chungyau97 <chungyau97@gmail.com>
2025-05-27 07:29:42 +01:00
Henry Heng 82d16458e4
Bugfix/Update file loader (#4420)
* update file loader

* delete temp for o1

* fix tavily
2025-05-14 10:30:33 +08:00
Henry Heng 7924fbce0d
Feature/agentflow v2 (#4298)
* agent flow v2

* chat message background

* conditon agent flow

* add sticky note

* update human input dynamic prompt

* add HTTP node

* add default tool icon

* fix export duplicate agentflow v2

* add agentflow v2 marketplaces

* refractor memoization, add iteration nodes

* add agentflow v2 templates

* add agentflow generator

* add migration scripts for mysql, mariadb, posrgres and fix date filters for executions

* update agentflow chat history config

* fix get all flows error after deletion and rename

* add previous nodes from parent node

* update generator prompt

* update run time state when using iteration nodes

* prevent looping connection, prevent duplication of start node, add executeflow node, add nodes agentflow, chat history variable

* update embed

* convert form input to string

* bump openai version

* add react rewards

* add prompt generator to prediction queue

* add array schema to overrideconfig

* UI touchup

* update embedded chat version

* fix node info dialog

* update start node and loop default iteration

* update UI fixes for agentflow v2

* fix async drop down

* add export import to agentflowsv2, executions, fix UI bugs

* add default empty object to flowlisttable

* add ability to share trace link publicly, allow MCP tool use for Agent and Assistant

* add runtime message length to variable, display conditions on UI

* fix array validation

* add ability to add knowledge from vector store and embeddings for agent

* add agent tool require human input

* add ephemeral memory to start node

* update agent flow node to show vs and embeddings icons

* feat: add import chat data functionality for AgentFlowV2

* feat: set chatMessage.executionId to null if not found in import JSON file or database

* fix: MariaDB execution migration script to utf8mb4_unicode_520_ci

---------

Co-authored-by: Ong Chung Yau <33013947+chungyau97@users.noreply.github.com>
Co-authored-by: chungyau97 <chungyau97@gmail.com>
2025-05-10 10:21:26 +08:00
Shibin Das 9d9135bed5
feat: add "Jira" document loader (#4247) (#4248)
* feat: add "Jira" document loader (#4247)

* chore: Remove console.log

* Update Jira.ts

---------

Co-authored-by: Henry Heng <henryheng@flowiseai.com>
2025-04-16 02:54:16 +08:00
Michael Vonrueden d7c0858424
feat: Enterprise Github (#4221) 2025-04-03 01:17:56 +08:00
Henry Heng 20a797d2e0
Feature/Add ability to create new doc store on upsert (#3965)
add ability to create new doc store on upsert, update firecrawl properties
2025-02-01 14:28:50 +00:00
Jérémy JOURDIN 24eb437bad
[Feature] improve CsvLoader & clean code (#3830)
* Improve CSV Loader

* Improve S3 Loaders

---------

Co-authored-by: Henry <hzj94@hotmail.com>
2025-01-14 16:47:04 +00:00
Henry Heng 9a417bdc95
Bugfix/update nodevm sandbox options, sanitize tablename (#3818)
* update nodevm sandbox options, sanitize tablename

* sanitize file name when getFileFromStorage
2025-01-07 15:26:25 +00:00
Henry Heng 7aead83d61
Feature/update json loader to have dynamic metadata (#3811)
update json loader to have dynamic metadata
2025-01-06 16:18:49 +00:00
Ajay Satish bfd677059e
feat. add epub support #3629 (#3668)
* feat. add epub support

* Add dependencies
2024-12-12 13:30:31 +00:00
Henry Heng 940c8fd3b0
Chore/LC v0.3 (#3517)
* bump langchain version to 0.3, upgrades on all chat models

* update all docs loader to have documents and text output options

* fix pnpm lock file
2024-11-28 11:06:12 +00:00
Henry Heng a2c36b4447
Feature/Add new doc store upsert and refresh API (#3556)
add new doc store upsert and refresh API
2024-11-25 15:47:13 +00:00
ZacharyQin b01772302e
feat: Add Custom Api Url Param for Self-hosted FireCrawl endpoint (#3480)
* feat: Add Custom api url for Self-hoseted FireCrawl endpoint

* Update FireCrawlApi.credential.ts

---------

Co-authored-by: Henry Heng <henryheng@flowiseai.com>
2024-11-09 00:44:17 +00:00
Ryno ccc0088d0d
feat: Add BraveSearchAPI document loader (#3486) 2024-11-08 12:59:22 +00:00
Jérémy JOURDIN a159da5bfe
[Usntructured loaders] Make Unstructured API URL optional when environment variable is present (#3414)
* Make Unstructured API URL optional when environment variable is present

* Fix empty apiUrl option in Unsctructured flowise loader

* Add focumentation for env vars
2024-11-01 22:03:53 +00:00
Henry Heng 53e504c32f
Feature/Full File Uploads & Message Delete API (#3314)
* add functionality for full file uploads, add remove messages from view dialog and API

* add attachments swagger

* update question to include uploadedFilesContent

* make config dialog modal lg size
2024-10-23 11:00:46 +01:00
Henry Heng 7d88183eca
Feature/add ability to specify dynamic metadata to jsonlines (#3238)
* add ability to specify dynamic metadata to jsonlines

* fix additional metadata
2024-09-23 20:45:48 +01:00
cragwolfe 7bdc274e93
chore: update unstructured API url and doc reference (#3224)
chore: udpate unstructured API url and doc reference
2024-09-20 17:17:05 +01:00
Henry Heng 7a4c7efcab
Chore/changed vm2 to @flowiseai/nodevm (#3200)
* changed vm2 to @flowiseai/nodevm

* update set-value and lunary version
2024-09-17 01:57:02 +01:00
Rafael Reis 43f11f21f8
Enhance Airtable Document Loader with Filter and Text Output (#3074) 2024-08-25 13:26:39 +01:00
Henry Heng 66acd0c000
Feature/add ability to upload file from chat (#3059)
add ability to upload file from chat
2024-08-25 13:22:48 +01:00
William Espegren 99ef9214c7
feat: additional metadata added to Spider tool (#2923)
* feat: additional metadata added to Spider tool

* console.log error instead of throwing to keep flow

* add omit

* pnpm lint
2024-08-02 22:48:48 +01:00
Henry Heng 9035509050
Bugfix/update loaders to community package (#2843)
update loaders to community package
2024-07-22 18:39:37 +01:00
Henry Heng bca4de0c63
Feature/seq agents (#2798)
* update build functions

* sequential agents

* update langchain to 0.2, added sequential agent nodes

* add marketplace templates

* update howto wordings

* Merge branch 'main' into feature/Seq-Agents

# Conflicts:
#	pnpm-lock.yaml

* update deprecated functions and add new sequential nodes

* add marketplace templates

* update marketplace templates, add structured output to llm node

* add multi agents template

* update llm node with bindmodels

* update cypress version

* update templates sticky note wordings

* update tool node to include human in loop action

* update structured outputs error from models

* update cohere package to resolve google genai pipeThrough bug

* update mistral package version, added message reconstruction before invoke seq agent

* add HITL to agent

* update state messages restructuring

* update load and split methods for s3 directory
2024-07-22 17:46:14 +01:00
slaplante-raft 34d0e4302c
S3 Directory Document Loading Component (#2818)
* Add new S3Directory component

* Add Additional Metadata and Omit Metadata Keys parameters

* Update S3Directory.ts

add placeholder for prefix

---------

Co-authored-by: Scott Laplante <scott.laplante@Scotts-MacBook-Pro.local>
Co-authored-by: Henry Heng <henryheng@flowiseai.com>
2024-07-22 00:06:48 +01:00
Henry Heng 363d1bfc44
Chore/update deprecating nodes (#2540)
* update deprecating nodes

* add filters use cases to marketplace

* update log level
2024-07-12 18:37:57 +01:00
Ahmed Osman 90558ca688
FIX #2617 Cherio Web Crawler doesn't work with large sites (#2678)
* FIX #2617 Big sites scan error

* FIX #2617 Big sites scan error - review fix

---------

Co-authored-by: Ahmed Osman <ahmed.osman@evolpe.pl>
2024-07-05 11:34:47 +01:00
William Espegren cacbfa8162
feat: Add limit parameter to Spider tool (#2762)
* feat: Add limit parameter to Spider tool

* fix pnpm lint
2024-07-05 11:23:34 +01:00
William Espegren 656f6cad81
Feature/Spider (open-source web scraper & crawler) (#2738)
* Add Spider Scraper & Crawler

* fix pnpm lint

* chore: Update metadata to be correct format

* fix pnpm lint
2024-07-02 00:00:52 +01:00
Henry Heng b55f87cc40
Feature/FireCrawl (#2728)
* add firecrawl

* Update FireCrawl.ts (#2692)

---------

Co-authored-by: Eric Ciarla <43451761+ericciarla@users.noreply.github.com>
2024-06-26 14:40:43 +01:00
Mohamed Akram c34eb8ee15
Unstructured Upsert bug (#2628)
* Unstructured Upsert bug
When upserting with the API, the uploaded files are of type pdfFile, txtFile, etc.
but the code reads only fileObject which is the uploaded file using the button

* Update UnstructuredFile.ts

fixed linting error

---------

Co-authored-by: Mohamed Akram <makram@ntgclarity.com>
2024-06-14 02:39:46 +01:00
Henry Heng 8ebc4dcfd5
Feature/lang graph (#2319)
* add langgraph

* datasource: initial commit

* datasource: datasource details and chunks

* datasource: Document Store Node

* more changes

* Document Store - Base functionality

* Document Store Loader Component

* Document Store Loader Component

* before merging the modularity PR

* after merging the modularity PR

* preview mode

* initial draft PR

* fixes

* minor updates and  fixes

* preview with loader and splitter

* preview with credential

* show stored chunks

* preview update...

* edit config

* save, preview and other changes

* save, preview and other changes

* save, process and other changes

* save, process and other changes

* alpha1 - for internal testing

* rerouting urls

* bug fix on new leader create

* pagination support for chunks

* delete document store

* Update pnpm-lock.yaml

* doc store card view

* Update store files to use updated storage functions, Document Store Table View and other changes

* ui changes

* add expanded chunk dialog, improve ui

* change throw Error to InternalError

* Bug Fixes and removal of subFolder, adding of view chunks for store

* lint fixes

* merge changes

* DocumentStoreStatus component

* ui changes for doc store

* add remove metadata key field, add custom document loader

* add chatflows used doc store chips

* add types/interfaces to DocumentStore Services

* document loader list dialog title bar color change

* update interfaces

* Whereused Chatflow Name and Added chunkNo to retain order of created chunks.

* use typeorm order chunkNo, ui changes

* update tabler icons react

* cleanup agents

* add pysandbox tool

* add abort functionality, loading next agent

* add empty view svg

* update chatflow tool with chatId

* rename to agentflows

* update worker for prompt input values

* update dashboard to agentflows, agentcanvas

* fix marketplace use template

* add agentflow templates

* resolve merge conflict

* update baseURL

---------

Co-authored-by: vinodkiran <vinodkiran@usa.net>
Co-authored-by: Vinod Paidimarry <vinodkiran@outlook.in>
2024-05-21 16:36:42 +01:00
Vinod Kiran 95f1090bed
BugFix #2386: Double quotes are not escaped, flow crashes (#2448)
Fix for #2386
2024-05-21 12:10:30 +01:00
Henry Heng b50103021c
Feature/Ability to omit all metadata keys using asterisk (#2401)
add ability to omit all metadata keys using asterisk
2024-05-13 16:30:57 +01:00
automaton82 43b22476e3
Fixes 2343 CSV error with no text splitter (#2344) 2024-05-07 01:43:42 +01:00
Vinod Kiran 40e36d1b39
Feature/DocumentStore (#2106)
* datasource: initial commit

* datasource: datasource details and chunks

* datasource: Document Store Node

* more changes

* Document Store - Base functionality

* Document Store Loader Component

* Document Store Loader Component

* before merging the modularity PR

* after merging the modularity PR

* preview mode

* initial draft PR

* fixes

* minor updates and  fixes

* preview with loader and splitter

* preview with credential

* show stored chunks

* preview update...

* edit config

* save, preview and other changes

* save, preview and other changes

* save, process and other changes

* save, process and other changes

* alpha1 - for internal testing

* rerouting urls

* bug fix on new leader create

* pagination support for chunks

* delete document store

* Update pnpm-lock.yaml

* doc store card view

* Update store files to use updated storage functions, Document Store Table View and other changes

* ui changes

* add expanded chunk dialog, improve ui

* change throw Error to InternalError

* Bug Fixes and removal of subFolder, adding of view chunks for store

* lint fixes

* merge changes

* DocumentStoreStatus component

* ui changes for doc store

* add remove metadata key field, add custom document loader

* add chatflows used doc store chips

* add types/interfaces to DocumentStore Services

* document loader list dialog title bar color change

* update interfaces

* Whereused Chatflow Name and Added chunkNo to retain order of created chunks.

* use typeorm order chunkNo, ui changes

---------

Co-authored-by: Henry <hzj94@hotmail.com>
Co-authored-by: Henry Heng <henryheng@flowiseai.com>
2024-05-06 15:23:27 +01:00
Vinod Kiran d5a97060e2
FEATURE: Adding File Upload to Unstructured Loader (#2304)
* initial commit

* updates to loader to support file upload

* updates to loader to support file upload

* update unstructured file

---------

Co-authored-by: Henry <hzj94@hotmail.com>
2024-05-02 18:34:32 +01:00
Henry Heng 2b1273ca31
Bugfix/Missing Filter for VectorStore to Document (#2285)
add filter for vector store to document
2024-04-29 22:25:40 +01:00
Vinod Kiran 7006d64de0
Feature/s3 storage (#2226)
* centralizing file writing....

* allowing s3 as storage option

* allowing s3 as storage option

* update s3 storage

---------

Co-authored-by: Henry <hzj94@hotmail.com>
2024-04-23 11:35:38 +01:00
Quinn 4c2ba109fd
Update unstructured document loaders (#2213)
* Update UnstructuredFile with missing values. Removed deprecated values.

* Update UnstructuredFolder with missing values. Removed deprecated values.

* Added suport for sourceIdKey to unstructured loaders

* Update unstructured hi_res model names

* Update S3File document loader with latest unstructured and model changes

* Update credential method for S3File document loader

* moved pnpm req to engines to avoid minor version changes

* Change unstructured skipInferTableTypes parse to JSON

* Update unstructured with new params. Also fixed list order, missing values, and support for null on multiOptions.
2024-04-21 19:42:28 +01:00
louyongjiu f5be889ea8
Feature: Add pdfUsage parameter setting support to folderFiles (#2211)
* Add pdfUsage parameter setting support to folderFiles

* Add pdfUsage parameter setting additionalParams: true
2024-04-19 11:43:22 +01:00
Vinod Kiran 658fa3984e
Feature/externalize files from chatflow - do not save as base64 (#1976)
* initial commit. Externalizing the file base64 string from flowData

* csv - docloader - Externalizing the file base64 string from flowData

* csv - docloader - Externalizing the file base64 string from flowData

* DocX - docloader - Externalizing the file base64 string from flowData

* Json - docloader - Externalizing the file base64 string from flowData

* Jsonlines - docloader - Externalizing the file base64 string from flowData

* PDF - docloader - Externalizing the file base64 string from flowData

* Vectara - vector store - Externalizing the file base64 string from flowData

* OpenAPIToolkit - tools - Externalizing the file base64 string from flowData

* OpenAPIChain - chain - Externalizing the file base64 string from flowData

* lint fixes

* datasource enabled - initial commit

* CSVAgent - agents - Externalizing the file base64 string from flowData

* Externalizing the file base64 string from flowData

* Externalizing the file base64 string from flowData

* add pnpm-lock.yaml

* update filerepository to add try catch

* Rename FileRepository.ts to fileRepository.ts

---------

Co-authored-by: Henry <hzj94@hotmail.com>
Co-authored-by: Henry Heng <henryheng@flowiseai.com>
2024-04-04 21:41:06 +05:30
Henry Heng b9b0c9d227
Bugfix/Web Scraper Limit (#2083)
fix when limit set to 0, selectedLinks sliced to become empty
2024-04-02 11:14:04 +01:00
Mr Khachaturov 4ca82ee733
Extend Confluence Document Loader to Support Server/Data Center with PAT (#1998)
* Extend Confluence Document Loader to Support Server/Data Center with PAT

- Update authentication to use Personal Access Token (PAT)
- Expand compatibility to include both Confluence Cloud and Server/Data Center

* Update ConfluenceServerDCApi.credential.ts

* use the same confluence loader with different connection logic

* use the same confluence loader with different connection logic

* Apply Prettier formatting
2024-03-25 01:37:55 +08:00
Henry 5a45a99620 Merge branch 'main' into chore/Upgrade-LC-version 2024-02-19 17:39:32 +08:00
Ilango 4e8bf4903d
Merge pull request #1687 from 0xi4o/bug/scrap-limit
Fix: relative links method and limit not applying to manage links
2024-02-12 12:35:40 +05:30
Ilango 5471a4c9aa Show error when relative links method is not set and allow 0 as limit value 2024-02-12 12:01:19 +05:30
Henry caf54bf31b add document json output 2024-02-09 16:07:34 +08:00
Henry 73112ad122 Merge branch 'main' into chore/Upgrade-LC-version
# Conflicts:
#	packages/components/package.json
2024-02-07 02:01:32 +08:00
Ilyes Tascou 19fb13baf0 fix for linting 2024-02-06 14:36:32 +01:00