Orange: Duplicate Detection - Revision history

Onnowpurbo: /* Contoh */

2020-04-13T02:11:44Z

Contoh

← Older revision		Revision as of 02:11, 13 April 2020
Line 30:		Line 30:
	Contoh sederhana ini menggunakan data iris untuk menemukan contoh data yang identik. Load iris dengan widget File dan meneruskannya ke widget Distances. Dalam widget Distances, gunakan Euclidean distance untuk menghitung distance matrix. Lewati widget Distances ke widget Duplicate Detection.		Contoh sederhana ini menggunakan data iris untuk menemukan contoh data yang identik. Load iris dengan widget File dan meneruskannya ke widget Distances. Dalam widget Distances, gunakan Euclidean distance untuk menghitung distance matrix. Lewati widget Distances ke widget Duplicate Detection.

	Sepertinya cluster C147 berisi tiga entri duplikat. Mari kita pilih di widget Duplicate Detection dan amati di widget ~~Tabel~~ Data. Ingatlah untuk mengatur output ke Duplicates Cluster. Tiga contoh data identik. Untuk menggunakan kumpulan data tanpa duplikat, gunakan output pertama, Corpus Tanpa Duplikat.		Sepertinya cluster C147 berisi tiga entri duplikat. Mari kita pilih di widget Duplicate Detection dan amati di widget Data Table. Ingatlah untuk mengatur output ke Duplicates Cluster. Tiga contoh data identik. Untuk menggunakan kumpulan data tanpa duplikat, gunakan output pertama, Corpus Tanpa Duplikat.

	Prosedur yang sama dapat digunakan juga untuk corpus. Ingatlah untuk menggunakan widget Bag of Words antara widget Corpus dan widget Distances.		Prosedur yang sama dapat digunakan juga untuk corpus. Ingatlah untuk menggunakan widget Bag of Words antara widget Corpus dan widget Distances.

	[[File:Duplicate-Detection-Example.png\|center\|600px\|thumb]]		[[File:Duplicate-Detection-Example.png\|center\|600px\|thumb]]


	==Youtube==		==Youtube==

Onnowpurbo: /* Output */

2020-04-13T01:53:39Z

Output

← Older revision		Revision as of 01:53, 13 April 2020
Line 18:		Line 18:
	Untuk men-set tingkat kesamaan, geser garis vertikal ke kiri atau kanan dalam visualisasi. Semakin jauh meninggalkan garis, semakin mirip dokumen harus dianggap duplikat. Kita juga dapat mengatur threshold secara manual di area kontrol.		Untuk men-set tingkat kesamaan, geser garis vertikal ke kiri atau kanan dalam visualisasi. Semakin jauh meninggalkan garis, semakin mirip dokumen harus dianggap duplikat. Kita juga dapat mengatur threshold secara manual di area kontrol.

	[[File:Duplicate-Detection-stamped.png\|center\|~~200px~~\|thumb]]		[[File:Duplicate-Detection-stamped.png\|center\|600px\|thumb]]

	* Information on unique and duplicate documents.		* Information on unique and duplicate documents.

Onnowpurbo: /* Referensi */

2020-04-11T22:26:16Z

Referensi

← Older revision		Revision as of 22:26, 11 April 2020
Line 36:		Line 36:
	[[File:Duplicate-Detection-Example.png\|center\|600px\|thumb]]		[[File:Duplicate-Detection-Example.png\|center\|600px\|thumb]]


			==Youtube==

			* [https://youtu.be/mC153JAyMIM YOUTUBE: ORANGE Duplicate Detection]

	==Referensi==		==Referensi==

Onnowpurbo at 13:36, 11 April 2020

2020-04-11T13:36:18Z

← Older revision		Revision as of 13:36, 11 April 2020
Line 2:		Line 2:


	~~Detect & remove duplicates from a~~ corpus.		Widget Duplicate Detection mendeteksi dan membuang semua duplikasi di corpus.

	==Input==		==Input==
Line 14:		Line 14:
	Corpus: Corpus with appended cluster labels.		Corpus: Corpus with appended cluster labels.

	Duplicate Detection ~~uses~~ clustering ~~to find duplicates in the~~ corpus. ~~It is great with the~~ Twitter ~~widget for removing retweets and other similar documents~~.		Widget Duplicate Detection menggunakan clustering untuk menemukan duplikat di corpus. Widget Duplicate Detection sangat membantu dengan widget Twitter untuk membuang retweet atau dokumen yang sama.

	To set ~~the level of similarity~~, ~~drag the line vertical line left or right in the visualization~~. ~~The further left the line~~, ~~the more similar the documents have to be in order to be considered duplicates~~. ~~You can also set the~~ threshold ~~manually in the control~~ area.		Untuk men-set tingkat kesamaan, geser garis vertikal ke kiri atau kanan dalam visualisasi. Semakin jauh meninggalkan garis, semakin mirip dokumen harus dianggap duplikat. Kita juga dapat mengatur threshold secara manual di area kontrol.

	[[File:Duplicate-Detection-stamped.png\|center\|200px\|thumb]]		[[File:Duplicate-Detection-stamped.png\|center\|200px\|thumb]]
Line 28:		Line 28:
	==Contoh==		==Contoh==

	~~This simple example uses~~ iris data ~~to find identical data instances~~. Load iris ~~with the~~ File widget ~~and pass it to~~ Distances. In Distances, ~~use~~ Euclidean distance ~~for computing the~~ distance matrix. ~~Pass distances to~~ Duplicate Detection.		Contoh sederhana ini menggunakan data iris untuk menemukan contoh data yang identik. Load iris dengan widget File dan meneruskannya ke widget Distances. Dalam widget Distances, gunakan Euclidean distance untuk menghitung distance matrix. Lewati widget Distances ke widget Duplicate Detection.

	~~It looks like~~ cluster C147 ~~contain three duplicate entries~~. ~~Let us select it in the~~ widget ~~and observe it in a~~ Data ~~Table~~. ~~Remember to set the~~ output to Duplicates Cluster. ~~IThe three~~ data ~~instances are identical~~. ~~To use the~~ data ~~set without duplicates~~, ~~use the first~~ output, Corpus ~~Without Duplicates~~.		Sepertinya cluster C147 berisi tiga entri duplikat. Mari kita pilih di widget Duplicate Detection dan amati di widget Tabel Data. Ingatlah untuk mengatur output ke Duplicates Cluster. Tiga contoh data identik. Untuk menggunakan kumpulan data tanpa duplikat, gunakan output pertama, Corpus Tanpa Duplikat.

	~~The same procedure can be used also for corpora~~. ~~Remember to use the~~ Bag of Words ~~between~~ Corpus ~~and~~ Distances.		Prosedur yang sama dapat digunakan juga untuk corpus. Ingatlah untuk menggunakan widget Bag of Words antara widget Corpus dan widget Distances.

	[[File:Duplicate-Detection-Example.png\|center\|~~200px~~\|thumb]]		[[File:Duplicate-Detection-Example.png\|center\|600px\|thumb]]

Onnowpurbo at 04:08, 29 January 2020

2020-01-29T04:08:48Z

Onnowpurbo at 09:37, 24 January 2020

2020-01-24T09:37:31Z

← Older revision		Revision as of 09:37, 24 January 2020
Line 18:		Line 18:
	To set the level of similarity, drag the line vertical line left or right in the visualization. The further left the line, the more similar the documents have to be in order to be considered duplicates. You can also set the threshold manually in the control area.		To set the level of similarity, drag the line vertical line left or right in the visualization. The further left the line, the more similar the documents have to be in order to be considered duplicates. You can also set the threshold manually in the control area.

	~~../_images/~~Duplicate-Detection-stamped.png		[[File:Duplicate-Detection-stamped.png\|center\|200px\|thumb]]

	Information on unique and duplicate documents.		Information on unique and duplicate documents.
Line 26:		Line 26:
	List of clusters at the selected threshold. They are sorted by size by default. Click on the cluster to observe its content on the output.		List of clusters at the selected threshold. They are sorted by size by default. Click on the cluster to observe its content on the output.

	~~Example~~		==Contoh==

	This simple example uses iris data to find identical data instances. Load iris with the File widget and pass it to Distances. In Distances, use Euclidean distance for computing the distance matrix. Pass distances to Duplicate Detection.		This simple example uses iris data to find identical data instances. Load iris with the File widget and pass it to Distances. In Distances, use Euclidean distance for computing the distance matrix. Pass distances to Duplicate Detection.
Line 34:		Line 34:
	The same procedure can be used also for corpora. Remember to use the Bag of Words between Corpus and Distances.		The same procedure can be used also for corpora. Remember to use the Bag of Words between Corpus and Distances.

	~~../_images/~~Duplicate-Detection-Example.png		[[File:Duplicate-Detection-Example.png\|center\|200px\|thumb]]

Onnowpurbo: Created page with "Sumber: https://orange3-text.readthedocs.io/en/latest/widgets/duplicatedetection.html Detect & remove duplicates from a corpus. Inputs Distances: A distance matrix. O..."

2020-01-19T09:50:28Z

Created page with "Sumber: https://orange3-text.readthedocs.io/en/latest/widgets/duplicatedetection.html Detect & remove duplicates from a corpus. Inputs Distances: A distance matrix. O..."

New page

Sumber: https://orange3-text.readthedocs.io/en/latest/widgets/duplicatedetection.html

Detect & remove duplicates from a corpus.

Inputs

Distances: A distance matrix.

Outputs

Corpus Without Duplicated: Corpus with duplicates removed.
Duplicates Cluster: Documents belonging to selected cluster.
Corpus: Corpus with appended cluster labels.

Duplicate Detection uses clustering to find duplicates in the corpus. It is great with the Twitter widget for removing retweets and other similar documents.

To set the level of similarity, drag the line vertical line left or right in the visualization. The further left the line, the more similar the documents have to be in order to be considered duplicates. You can also set the threshold manually in the control area.

../_images/Duplicate-Detection-stamped.png

Information on unique and duplicate documents.
Linkage used for clustering (Single, Average, Complete, Weighted and Ward).
Distance threshold sets the similarity cutoff. The lower the value, the more similar the data instances have to be to belong to the same cluster. You can also set the cutoff by dragging the vertical line in the plot.
Cluster labels can be appended as attributes, class or metas.
List of clusters at the selected threshold. They are sorted by size by default. Click on the cluster to observe its content on the output.

Example

This simple example uses iris data to find identical data instances. Load iris with the File widget and pass it to Distances. In Distances, use Euclidean distance for computing the distance matrix. Pass distances to Duplicate Detection.

It looks like cluster C147 contain three duplicate entries. Let us select it in the widget and observe it in a Data Table. Remember to set the output to Duplicates Cluster. IThe three data instances are identical. To use the data set without duplicates, use the first output, Corpus Without Duplicates.

The same procedure can be used also for corpora. Remember to use the Bag of Words between Corpus and Distances.

../_images/Duplicate-Detection-Example.png

==Referensi==

* https://orange3-text.readthedocs.io/en/latest/widgets/duplicatedetection.html

==Pranala Menarik==

* [[Orange]]

← Older revision		Revision as of 04:08, 29 January 2020
Line 4:		Line 4:
	Detect & remove duplicates from a corpus.		Detect & remove duplicates from a corpus.

	~~Inputs~~		==Input==

	Distances: A distance matrix.		Distances: A distance matrix.

	~~Outputs~~		==Output==

	Corpus Without Duplicated: Corpus with duplicates removed.		Corpus Without Duplicated: Corpus with duplicates removed.
	Duplicates Cluster: Documents belonging to selected cluster.		Duplicates Cluster: Documents belonging to selected cluster.
	Corpus: Corpus with appended cluster labels.		Corpus: Corpus with appended cluster labels.

	Duplicate Detection uses clustering to find duplicates in the corpus. It is great with the Twitter widget for removing retweets and other similar documents.		Duplicate Detection uses clustering to find duplicates in the corpus. It is great with the Twitter widget for removing retweets and other similar documents.
Line 20:		Line 20:
	[[File:Duplicate-Detection-stamped.png\|center\|200px\|thumb]]		[[File:Duplicate-Detection-stamped.png\|center\|200px\|thumb]]

	Information on unique and duplicate documents.		* Information on unique and duplicate documents.
	Linkage used for clustering (Single, Average, Complete, Weighted and Ward).		* Linkage used for clustering (Single, Average, Complete, Weighted and Ward).
	Distance threshold sets the similarity cutoff. The lower the value, the more similar the data instances have to be to belong to the same cluster. You can also set the cutoff by dragging the vertical line in the plot.		* Distance threshold sets the similarity cutoff. The lower the value, the more similar the data instances have to be to belong to the same cluster. You can also set the cutoff by dragging the vertical line in the plot.
	Cluster labels can be appended as attributes, class or metas.		* Cluster labels can be appended as attributes, class or metas.
	List of clusters at the selected threshold. They are sorted by size by default. Click on the cluster to observe its content on the output.		* List of clusters at the selected threshold. They are sorted by size by default. Click on the cluster to observe its content on the output.

	==Contoh==		==Contoh==